Electronic device and method of executing function of electronic device

ABSTRACT

An artificial intelligence system and method are disclosed herein. The system includes a processor which implements the method, including: receiving by an input unit a first user input including a request to execute a task using at least one of the electronic device or an external device, transmitting by a wireless communication unit first data associated with the first user input to an external server, receiving a first response from the external server including information associated with at least one of the first user input and a sequence of electronic device states for performing at least a portion of the task, receiving a second user input assigning at least one of a voice command and a touch operation received by a touch screen display as the request to perform the task, and transmitting second data associated with the second user input to the external server.

CROSS REFERENCE TO RELATED APPLICATION(S)

This application is a Continuation of U.S. patent application Ser. No.16/589,257 filed on Oct. 1, 2019 which claims the benefit of the earlierU.S. patent application Ser. No. 16/389,216 filed on Apr. 19, 2019 andassigned U.S. Pat. No. 10,440,167 issued on Oct. 8, 2019 which claimsthe benefit of the earlier U.S. patent application Ser. No. 15/911,329filed on Mar. 5, 2018 and assigned U.S. Pat. No. 10,547,729 issued onJan. 28, 2020 which claims the benefit of priority under 35 U.S.C. §119(a) to Korean Patent Application Serial No. 10-2017-0038722, whichwas filed in Korean Intellectual Property Office on Mar. 27, 2017 andKorean Patent Application Serial No. 10-2017-0066368, which was filed inthe Korean Intellectual Property Office on May 29, 2017, the entirecontent of which is hereby incorporated by reference.

TECHNICAL FIELD

The present disclosure relates to an electronic device and a method ofexecuting the function of the electronic device.

BACKGROUND

An artificial intelligence system is a computer system that attempts toimplement aspects of human-level intelligence, in which a machine canlearn and form judgements to improve a recognition rate for informationas it is used.

Artificial intelligence technologies include a machine learning (deeplearning) technology that uses an algorithm that classifies/learns thecharacteristics of input data by itself and an elemental technology thatsimulates functions such as recognition, judgment, and the like of humanbrain by utilizing a machine learning algorithm.

The elemental technology may include at least one of, for example, alinguistic comprehension technique for recognizing humanlanguages/characters, a visual comprehension technique for recognizingobjects as in human vision, a reasoning/predicting technique for judgingand logically reasoning and predicting information, a knowledgeexpression technique for processing human experience information asknowledge data, and an operation control technique for controllingautonomous driving of the vehicle or the motion of a robot.

For example, the linguistic comprehension technique is a technique thatrecognizes and applies/processes human languages/characters, and mayinclude natural language processing, machine translation, a dialoguesystem, query response, speech recognition/synthesis, and the like.

For example, the visual comprehension technique is a technique thatrecognizes and processes objects as in human vision, and may includeobject recognition, object tracking, image retrieval, human recognition,scene understanding, spatial understanding, image enhancement, and thelike.

For example, the reasoning/predicting technique is a technique thatjudges and logically reasons and predicts information, and may includeknowledge/probability-based reasoning, optimization prediction,preference-based planning, recommendation, and the like.

For example, the knowledge expression technique is a technique thatautomates the processing of human experience information into knowledgedata, and may include knowledge building (datageneration/classification), knowledge management (data utilization), andthe like.

For example, the operation control technique is a technique thatcontrols autonomous vehicle motion and robot motion, and may includemotion control (navigation, impulse, running), operation control(behavior control), and the like.

SUMMARY

When an electronic device provides a virtualized computer-implemented‘intelligent agent,’ a user may lack prior knowledge of the function ofthe intelligent agent and may also have difficulty in explicitlyrecognizing the intelligent agent. Accordingly, user learning for usingthe intelligent agent may be desirable, and the electronic device mayerroneously recognize a user's question, request, command, or the like,and may perform an operation different from the intention of the user.In addition, in order to easily use the function of the intelligentagent, the user may be provided with a hint to inform the user of avoice that the user can input so that the operation of the electronicdevice can be performed through a voice input.

A user unfamiliar with the use of the electronic device or unfamiliarwith performing the function of the electronic device through a voicecommand may have difficulty knowing what function to perform or whichutterance should be requested to perform the function, even in asituation where the specific function of the electronic device is to beexecuted. In addition, when a user wishes to execute any one function ofthe electronic device through the voice command, the user's utterancemay be related to the corresponding function to execute thecorresponding function.

According to various embodiments, there are provided with an electronicdevice that provides a hint that informs a user of a voice input thatthe user can input so that the operation of the electronic device can beperformed through the voice input, and a method of operating theelectronic device.

According to various embodiments, an electronic device includes: ahousing; a touch screen display that is exposed through a portion of thehousing; a wireless communication circuit; a microphone; a speaker; atleast one processor that is located inside the housing and iselectrically connected to the touch screen display, the wirelesscommunication circuit, the microphone, and the speaker; and a memorythat is located inside the housing and is electrically coupled to the atleast one processor, wherein the memory is configured to at leasttemporarily store a plurality of reference sequences of states and eachof the plurality of reference sequences of states is associated withindividual operations of the electronic device or individualcombinations of the operations, and when the memory is performed, the atleast one processor receives a first sequence of touch inputs from auser via the touch screen display, sequentially changes the states ofthe electronic device to perform the operations of the electronic deviceor the combinations of the operations in response to the first sequenceof the touch inputs, stores a first sequence of the changed states inthe memory, compares the first sequence of the changed states with atleast a portion of at least one of the plurality of reference sequencesof states, and stores instructions for causing at least one result fromthe comparison to be stored in the memory.

According to various embodiments, an electronic system includes: a touchscreen display; at least one communication circuit; a microphone; aspeaker; one or more processors that is operably connected to the touchscreen display, the communication circuit, the microphone, and thespeaker; and at least one memory that is operably connected to the oneor more processors, wherein the memory is configured to at leasttemporarily store a plurality of reference sequences of states, and eachof the plurality of reference sequences of states is associated withindividual operations of an electronic device or individual combinationsof the operations, and when the memory is performed, the one or moreprocessors acquires a first sequence of touch inputs from a user via thetouch screen display, sequentially changes the states of the electronicdevice to perform the operations of the electronic device or thecombinations of the operations in response to the first sequence of thetouch inputs, stores a first sequence of the changed states in thememory, compares the first sequence of the changed states with at leasta portion of at least one of the plurality of reference sequences ofstates, and stores instructions for causing at least one result from thecomparison to be stored in the memory.

According to various embodiments, an electronic device includes: ahousing; a touch screen display that is exposed through a portion of thehousing; a wireless communication circuit; a microphone; a speaker; atleast one processor that is located inside the housing and iselectrically connected to the touch screen display, the wirelesscommunication circuit, the microphone, and the speaker; and a memorythat is located inside the housing and is electrically connected to theat least one processor, wherein the memory is configured to at leasttemporarily store a plurality of reference sequences of states and theplurality of reference sequences of states is associated with individualoperations of the electronic device or individual combinations of theoperations, and when the memory is performed, the at least one processorreceives a first sequence of inputs from a user, sequentially changesthe states of the electronic device to perform the operations of theelectronic device or the combinations of the operations in response tothe first sequence of the inputs, stores a first sequence of the changedstates in the memory, compares the first sequence of the changed stateswith at least a portion of at least one of the plurality of referencesequences of states, and stores instructions for causing at least oneresult from the comparison to be stored in the memory.

According to various embodiments, an electronic device includes: ahousing; a touch screen display that is exposed through a portion of thehousing; a wireless communication circuit; a microphone; a speaker; atleast one processor that is located inside the housing and iselectrically connected to the touch screen display, the wirelesscommunication circuit, the microphone, and the speaker; and a memorythat is located inside the housing and is electrically connected to theat least one processor, wherein the memory is configured to at leasttemporarily store a plurality of reference sequences of states and theplurality of reference sequences of states is associated with individualoperations of the electronic device or individual combinations of theoperations, and when the memory is performed, the at least one processorreceives one or more touch inputs from a user via the touch screendisplay, sequentially changes the states of the electronic device toperform the operations of the electronic device or the combinations ofthe operations in response to the one or more touch inputs, stores afirst sequence of the changed states in the memory, compares the firstsequence of the changed states with at least a portion of at least oneof the plurality of reference sequences of states, and storesinstructions for causing at least one result from the comparison to bestored in the memory.

According to various embodiments, an electronic device includes: ahousing; a touch screen display that is exposed through a portion of thehousing; an input device; at least one processor that is located insidethe housing and is electrically connected to the touch screen displayand the input device; and a memory that is located inside the housingand is electrically connected to the at least one processor, whereinwhen the memory is performed, the at least one processor receives afirst sequence of inputs from a user via the input device, sequentiallychanges a first sequence of states of the electronic device to perform afirst operation of the electronic device in response to the firstsequence of the inputs, stores information associated with the firstsequence of the states in the memory, provides an indication via thetouch screen display at least based on the information associated withthe first sequence of the states stored in the memory, the indicationindicating availability of the first operation, and stores, when theindication is selected, instructions causing performance of the firstoperation without a sequential change of the first sequence of thestates.

According to various embodiments, in a machine-readable storage mediumin which a program for executing a method of generating a hint forexecuting an operation of an electronic device is recorded, the methodincludes: receiving a first sequence of inputs from a user; sequentiallychanging a first sequence of states of the electronic device to performa first operation of the electronic device in response to the firstsequence of the inputs; storing information associated with the firstsequence of the states; providing an indication via a touch screendisplay of the electronic device at least based on the storedinformation associated with the first sequence of the states, whereinthe indication indicates availability of the first operation; andperforming the first operation without a sequential change of the firstsequence of the states when the indication is selected.

According to various embodiments, an electronic device includes: ahousing; a touch screen display that is located inside the housing andis exposed through a first portion of the housing; a microphone that islocated inside the housing and is exposed through a second portion ofthe housing; at least one speaker that is located inside the housing andis exposed through a third portion of the housing; a wirelesscommunication circuit that is located inside the housing; a processorthat is located inside the housing and is electrically connected to thetouch screen display, the microphone, the speaker, and the wirelesscommunication circuit; and a memory that is located inside the housingand is electrically connected to the processor, wherein when the memoryis performed, the processor receives a first user input via at least oneof the touch screen display and the microphone, the first user inputincluding a request to perform a task using at least one of theelectronic device and an external device, transmits first dataassociated with the first user input to an external server via thewireless communication circuit, receives a first response from theexternal server via the wireless communication circuit, the firstresponse including information associated with the first user inputand/or sequence information of states of the electronic device forperforming at least a portion of the task, receives a second user inputfor assigning an utterance and/or a user operation via the touch screendisplay as a request to perform the task, and stores instructionscausing to transmit second data associated with the second user input tothe external server via the communication circuit.

According to various embodiments, a method of executing an operation ofan electronic device includes: receiving a first user input, wherein thefirst user input includes a request to perform a task using at least oneof the electronic device or an external device; transmitting first dataassociated with the first user input to an external server; receiving afirst response from the external server, wherein the first responseincludes information associated with the first user input and/orsequence information of states of the electronic device for performingat least a portion of the task; receiving a second user input forassigning an utterance and/or a user operation via a touch screendisplay of the electronic device as the request to perform the task; andtransmitting second data associated with the second user input to theexternal server.

According to various embodiments, a machine-readable storage mediumrecording a program to perform a method of executing an operation of anelectronic device, the method comprising: receiving a first user input,wherein the first user input includes a request to perform a task usingat least one of the electronic device or an external device;transmitting first data associated with the first user input to anexternal server; receiving a first response from the external server,wherein the first response includes information associated with thefirst user input and/or sequence information of states of the electronicdevice for performing at least a portion of the task; receiving a seconduser input for assigning an utterance and/or a user operation via atouch screen display of the electronic device as the request to performthe task; and transmitting second data associated with the second userinput to the external server.

According to various embodiments, an electronic device includes: ahousing; a touch screen display that is located inside the housing andis exposed through a first portion of the housing; a microphone that islocated inside the housing and is exposed through a second portion ofthe housing; at least one speaker that is located inside the housing andis exposed through a third portion of the housing; a wirelesscommunication circuit that is located inside the housing; a processorthat is located inside the housing and is electrically connected to thetouch screen display, the microphone, the speaker, and the wirelesscommunication circuit; and a memory that is located inside the housingand is electrically connected to the processor, wherein the memorystores instructions that, when executed, cause the processor to: receivea first user input via at least one of the touch screen display and themicrophone, the first user input including a request to perform a taskusing at least one of the electronic device and an external device,transmit first data associated with the first user input to an externalserver via the wireless communication circuit, acquire a first responsebased on the first data associated with the first user input, the firstresponse including information associated with the first user inputand/or sequence information of states of the electronic device forperforming at least a portion of the task, receive a second user inputfor assigning an utterance and/or a user operation via the touch screendisplay as the request to perform the task, and acquire a secondresponse based on second data associated with the second user input viathe communication circuit, the second response storing instructionsincluding the sequence information of the states of the electronicdevice for performing the at least a portion of the task.

As described above, according to various embodiments, the electronicdevice may integrate user's control histories (control experience) forthe electronic device to provide a hint to a user, so that the user maycontrol the electronic device in various ways.

In addition, the electronic device may provide a hint that is highlyusable in accordance with an occasion of the user and/or a utilizationpattern of a specific function of the user, thereby increasing the hintutilization of the user.

Further, the function of the electronic device may be executed using apersonalized dialog for a user.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and advantages of the presentdisclosure will be more apparent from the following detailed descriptiontaken in conjunction with the accompanying drawings, in which:

FIG. 1 illustrates a network environment including an electronic deviceaccording to various embodiments;

FIG. 2 is a block diagram illustrating an electronic device according tovarious embodiments;

FIG. 3 is a block diagram illustrating a program module according tovarious embodiments;

FIG. 4 is a flowchart illustrating a hint generation operation forexecuting an electronic device according to various embodiments;

FIG. 5 is a diagram illustrating a screen for providing a hint for theoperation of an electronic device according to various embodiments;

FIG. 6 is a flowchart illustrating a hint generation operation forexecuting an electronic device according to various embodiments;

FIG. 7 is a flowchart illustrating the operation of an electronic devicethat generates a path rule through user log collection of the electronicdevice according to various embodiments;

FIG. 8A, FIG. 8B, FIG. 8C and FIG. 8D are diagrams illustrating anoperation of determining a path rule of an electronic device dependingon a user input according to various embodiments;

FIG. 9 is a flowchart illustrating the operation of an electronic devicethat generates a path rule through user log collection of the electronicdevice according to various embodiments;

FIG. 10 is a flowchart illustrating the operation of an electronicdevice that generates a path rule through user log collection of theelectronic device according to various embodiments;

FIG. 11 is a flowchart illustrating a hint generation operation forexecution of an electronic device according to various embodiments;

FIG. 12 is a flowchart illustrating the operation of an electronicdevice that generates a path rule through user log collection of theelectronic device according to various embodiments;

FIG. 13 is a flowchart illustrating the operation of an electronicdevice that generates a path rule through user log collection of theelectronic device according to various embodiments;

FIG. 14 is a flowchart illustrating the operation of an electronicdevice that generates a path rule through user log collection of theelectronic device according to various embodiments;

FIG. 15 is a flowchart illustrating the operation of an electronicdevice that generates a path rule through user log collection of theelectronic device according to various embodiments;

FIG. 16 is a flowchart illustrating the operation of an electronicdevice that generates a path rule through user log collection of theelectronic device according to various embodiments;

FIG. 17 is a diagram illustrating a hint table corresponding to a pathrule of an electronic device according to various embodiments;

FIGS. 18 to 20 are block diagrams illustrating a system for providing ahint;

FIG. 21 is a diagram illustrating a system that utilizes a private cloudto provide a personalized hint according to various embodiments;

FIG. 22 is a block diagram illustrating a system for providing a hintaccording to various embodiments;

FIG. 23 is a flowchart illustrating an operation for hint generation ofa server according to various embodiments;

FIG. 24 is a graph illustrating a type of a hint to be provided inconsideration of a use time of hint providing of an electronic deviceaccording to various embodiments and a hint exposure frequency thereof;

FIG. 25 is a diagram illustrating an integrated intelligent systemaccording to various embodiments of the present disclosure;

FIG. 26 is a block diagram illustrating a user terminal of an integratedintelligent system according to an embodiment of the present disclosure;

FIG. 27 is a diagram illustrating the execution of an intelligent app ofa user terminal according to an embodiment of the present disclosure;

FIG. 28 is a block diagram illustrating an intelligent server of anintegrated intelligent system according to an embodiment of the presentdisclosure;

FIG. 29 is a diagram illustrating a method of generating a path rule ofa path planner module according to an embodiment of the presentdisclosure;

FIG. 30 is a diagram illustrating a case in which a context module of aprocessor according to an embodiment of the present disclosure collectscurrent states;

FIG. 31 is a diagram illustrating a case in which a persona moduleaccording to an embodiment of the present disclosure manages userinformation;

FIG. 32 is a block diagram illustrating a suggestion module according toan embodiment of the present disclosure;

FIG. 33 is a diagram illustrating an example in which an electronicdevice according to an embodiment of the present disclosure provides ahint;

FIG. 34 is a diagram illustrating a system for executing the function ofan electronic device according to various embodiments;

FIG. 35 is a flowchart illustrating the operation of an electronicdevice for executing the function of the electronic device according tovarious embodiments;

FIG. 36 is a flowchart illustrating a macro setting operation forexecuting the function of an electronic device according to variousembodiments;

FIG. 37A and FIG. 37B are flowcharts illustrating an operation ofexecuting the function of an electronic device according to variousembodiments;

FIG. 38 is a flowchart illustrating an operation of executing thefunction of an electronic device according to various embodiments;

FIG. 39 is a diagram illustrating a system for executing the function ofan electronic device according to various embodiments;

FIG. 40 is a flowchart illustrating an operation of executing thefunction of an electronic device according to various embodiments;

FIG. 41 is a flowchart illustrating an operation of executing thefunction of an electronic device according to various embodiments;

FIG. 42 is a flowchart illustrating an operation of executing thefunction of an electronic device according to various embodiments;

FIG. 43 is a flowchart illustrating an operation of executing thefunction of an electronic device according to various embodiments;

FIG. 44A, FIG. 44B, and FIG. 44C are diagrams illustrating theregistration of a command shortcut for executing the function of anelectronic device according to various embodiments;

FIG. 45A and FIG. 45B are diagrams illustrating the registration of acommand shortcut for executing the function of an electronic deviceaccording to various embodiments;

FIG. 46A, FIG. 46B and FIG. 46C are diagrams illustrating theregistration of a command shortcut for executing the function of anelectronic device according to various embodiments;

FIG. 47A, FIG. 47B and FIG. 47C are diagrams illustrating theregistration of a command shortcut for executing the function of anelectronic device according to various embodiments; and

FIG. 48 is a diagram illustrating a command shortcut execution screen ofan electronic device according to various embodiments.

DETAILED DESCRIPTION

Hereinafter, various embodiments of the present disclosure will bedescribed with reference to the accompanying drawings. The embodimentsand the terms used therein are not intended to limit the technologydisclosed herein to specific forms, and should be understood to includevarious modifications, equivalents, and/or alternatives to thecorresponding embodiments. In describing the drawings, similar referencenumerals may be used to designate similar constituent elements. Asingular expression may include a plural expression unless they aredefinitely different in a context. As used herein, singular forms mayinclude plural forms as well unless the context clearly indicatesotherwise. The expression “a first”, “a second”, “the first”, or “thesecond” used in various embodiments of the present disclosure may modifyvarious components regardless of the order and/or the importance butdoes not limit the corresponding components. When an element (e.g.,first element) is referred to as being “(functionally orcommunicatively) connected,” or “directly coupled” to another element(second element), the element may be connected directly to the anotherelement or connected to the another element through yet another element(e.g., third element).

The expression “configured to” as used in various embodiments of thepresent disclosure may be interchangeably used with, for example,“suitable for”, “having the capacity to”, “designed to”, “adapted to”,“made to”, or “capable of” in terms of hardware or software, accordingto circumstances. Alternatively, in some situations, the expression“device configured to” may mean that the device, together with otherdevices or components, “is able to”. For example, the phrase “processoradapted (or configured) to perform A, B, and C” may mean a dedicatedprocessor (e.g. embedded processor) for performing the correspondingoperations or a generic-purpose processor (e.g., Central Processing Unit(CPU) or Application Processor (AP)) that can perform the correspondingoperations by executing one or more software programs stored in a memorydevice.

An electronic device according to various embodiments of the presentdisclosure may include at least one of, for example, a smart phone, atablet Personal Computer (PC), a mobile phone, a video phone, anelectronic book reader (e-book reader), a desktop PC, a laptop PC, anetbook computer, a workstation, a server, a Personal Digital Assistant(PDA), a Portable Multimedia Player (PMP), a MPEG-1 audio layer-3 (MP3)player, a mobile medical device, a camera, and a wearable device.According to various embodiments, the wearable device may include atleast one of an accessory type (e.g., a watch, a ring, a bracelet, ananklet, a necklace, a glasses, a contact lens, or a Head-Mounted Device(HMD)), a fabric or clothing integrated type (e.g., an electronicclothing), a body-mounted type (e.g., a skin pad, or tattoo), and abio-implantable type (e.g., an implantable circuit). In someembodiments, the electronic device may include at least one of, forexample, a television, a Digital Video Disk (DVD) player, an audio, arefrigerator, an air conditioner, a vacuum cleaner, an oven, a microwaveoven, a washing machine, an air cleaner, a set-top box, a homeautomation control panel, a security control panel, a TV box (e.g.,Samsung HomeSync™, Apple TV™, or Google TV™), a game console (e.g., Xboxand PlayStation™), an electronic dictionary, an electronic key, acamcorder, and an electronic photo frame.

In other embodiments, the electronic device may include at least one ofvarious medical devices (e.g., various portable medical measuringdevices (a blood glucose monitoring device, a heart rate monitoringdevice, a blood pressure measuring device, a body temperature measuringdevice, etc.), a Magnetic Resonance Angiography (MRA), a MagneticResonance Imaging (MRI), a Computed Tomography (CT) machine, and anultrasonic machine), a navigation device, a Global Positioning System(GPS) receiver, an Event Data Recorder (EDR), a Flight Data Recorder(FDR), a Vehicle Infotainment Devices, an electronic devices for a ship(e.g., a navigation device for a ship, and a gyro-compass), avionics,security devices, an automotive head unit, a robot for home or industry,an Automatic Teller's Machine (ATM) in banks, Point Of Sales (POS) in ashop, or internet device of things (e.g., a light bulb, various sensors,electric or gas meter, a sprinkler device, a fire alarm, a thermostat, astreetlamp, a toaster, a sporting goods, a hot water tank, a heater, aboiler, etc.). According to some embodiments, an electronic device mayinclude at least one of a part of furniture or a building/structure, anelectronic board, an electronic signature receiving device, a projector,and various types of measuring instruments (e.g., a water meter, anelectric meter, a gas meter, a radio wave meter, and the like). Invarious embodiments, the electronic device may be flexible, or may be acombination of one or more of the aforementioned various devices. Theelectronic device according to one embodiment of the present disclosureis not limited to the above described devices. In the presentdisclosure, the term “user” may indicate a person using an electronicdevice or a device (e.g., an artificial intelligence electronic device)using an electronic device.

Referring to FIG. 1 , in various embodiments, an electronic device 101in a network environment 100 is described. The electronic device 101 mayinclude a bus 110, a processor 120, a memory 130, an input/outputinterface 150, a display 160, and a communication interface 170implementing a network connection 164. In some embodiments, theelectronic device 101 may omit at least one of the components or furtherinclude other components. The bus 110 may include a circuit forconnecting the components 110 to 170 and transmitting communicationbetween the components (e.g., control messages or data). The processor120 may include one or more of a central processing unit (CPU), anapplication processor (AP), and a communication processor (CP). Forexample, the processor 120 may carry out operations or data processingrelated to control and/or communication of at least one other componentof the electronic device 101.

The memory 130 may include a volatile memory and/or a non-volatilememory. The memory 130 may store, for example, instructions or datarelevant to at least one other component of the electronic device 101.According to an embodiment, the memory 130 may store software and/or aprogram 140. The program 140 may include, for example, a kernel 141,middleware 143, an application programming interface (API) 145, and/orapplication programs (or “applications”) 147. At least some of thekernel 141, the middleware 143, and the API 145 may be referred to as anoperating system (OS). The kernel 141 may control or manage systemresources (e.g., the bus 110, the processor 120, or the memory 130) usedfor performing an operation or function implemented by the otherprograms (e.g., the middleware 143, the API 145, or the applicationprograms 147). Furthermore, the kernel 141 may provide an interfacethrough which the middleware 143, the API 145, or the applicationprograms 147 may access the individual components of the electronicdevice 101 to control or manage the system resources.

The middleware 143, for example, may function as an intermediary forallowing the API 145 or the application programs 147 to communicate withthe kernel 141 to exchange data. In addition, the middleware 143 mayprocess one or more task requests received from the application programs147 according to priorities thereof. For example, the middleware 143 mayassign, to at least one of the application programs 147, priorities forusing the system resources (e.g., the bus 110, the processor 120, thememory 130, and the like) of the electronic device 101, and may processthe one or more task requests. The API 145 is an interface through whichthe applications 147 control functions provided from the kernel 141 orthe middleware 143, and may include, for example, at least one interfaceor function (e.g., an instruction) for file control, window control,image processing, or text control.

The input/output interface 150 may transfer, for example, instructionsor data input from a user or another external device to the othercomponent(s) of the electronic device 101, or may output instructions ordata received from the other element(s) of the electronic device 101 toa user or another external device.

Examples of the display 160 may include a liquid crystal display (LCD),a light-emitting diode (LED) display, an organic LED (OLED) display, amicroelectromechanical system (MEMS) display, and an electronic paperdisplay. The display 160 may display, for example, various types ofcontents (e.g., text, images, videos, icons, and/or symbols) to users.The display 160 may include a touch screen, and may receive, forexample, a touch, gesture, proximity, or hovering input by using anelectronic pen or a part of the user's body. The communication interface170, for example, may set communication between the electronic device101 and an external device (e.g., a first external electronic device102, a second external electronic device 104, or a server 106). Forexample, the communication interface 170 may be connected to a network162 through wireless or wired communication to communicate with theexternal device (e.g., the second external electronic device 104 or theserver 106).

The wireless communication may include cellular communication that usesat least one of, for example, long term evolution (LTE), LTE-advanced(LTE-A), code division multiple access (CDMA), wideband CDMA (WCDMA),universal mobile telecommunications system (UMTS), wireless broadband(WiBro), and global system for mobile communications (GSM). According toan embodiment, the wireless communication may include, for example, atleast one of wireless fidelity (Wi-Fi), Bluetooth, Bluetooth low energy(BLE), Zigbee, near field communication (NFC), a magnetic securetransmission, radio frequency (RF), and body area network (BAN).According to an embodiment, the wireless communication may includeglobal navigation satellite system (GNSS). The GNSS may be, for example,a global positioning system (GPS), a global navigation satellite system(GLONASS), a BeiDou Navigation Satellite System (hereinafter referred toas “BeiDou”), or an European global satellite-based navigation system(Galileo). Hereinafter, in this specification, the “GPS” may beinterchangeably used with the “GNSS”. The wired communication mayinclude at least one of, for example, a universal serial bus (USB), ahigh definition multimedia interface (HDMI), recommended standard 232(RS-232), power line communication, and a plain old telephone service(POTS). The network 162 may include at least one of telecommunicationsnetworks, for example, a computer network (e.g., LAN or WAN), anInternet, or a telephone network.

Each of the first external electronic device 102 and the second externalelectronic device 104 may be of a type that is identical to, ordifferent from, that of the electronic device 101. According to variousembodiments, all or some of the operations performed in the electronicdevice 101 may be performed in another electronic device or a pluralityof electronic devices (e.g., the first external electronic device 102 orthe second external electronic device 104, or the server 106). Accordingto an embodiment, when the electronic device 101 has to perform somefunctions or services automatically or by request, the electronic device101 may make a request to perform at least some functions relatingthereto to another device (e.g., the first external electronic device102 or the second external electronic device 104, or the server 106)instead of, or in addition to, performing the functions or services byitself. Another electronic device (e.g., the first external electronicdevice 102 or the second external electronic device 104, or the server106) may execute the requested functions or the additional functions,and may deliver a result of the execution to the electronic device 101.The electronic device 101 may process the received result as it is oradditionally, and may provide the requested functions or services. Tothis end, for example, cloud computing, distributed computing, orclient-server computing technology may be used.

FIG. 2 is a block diagram illustrating an electronic device 201according to various embodiments. The electronic device 201 may include,for example, a part or the entirety of the electronic device 101illustrated in FIG. 1 . The electronic device 201 may include at leastone processor 210 (e.g., an AP), a communication module 220, asubscriber identification module (SIM) 224, a memory 230, a sensormodule 240, an input device 250, a display 260, an interface 270, anaudio module 280, a camera module 291, a power management module 295, abattery 296, an indicator 297, and a motor 298. The processor 210 maycontrol a plurality of hardware or software components connected to theprocessor 210 by driving, for example, an operating system or anapplication program, and may perform various data processing andcalculations. The processor 210 may be embodied as, for example, asystem on chip (SoC). According to an embodiment, the processor 210 mayfurther include a graphics processing unit (GPU) and/or an image signalprocessor. The processor 210 may include at least some (e.g., a cellularmodule 221) of the components illustrated in FIG. 2 . The processor 210may load, into a volatile memory, instructions or data received from atleast one (e.g., a non-volatile memory) of the other components and mayprocess the loaded instructions or data, and may store resultant data ina non-volatile memory.

The communication module 220 may have a configuration that is equal orsimilar to that of the communication interface 170 of FIG. 1 . Thecommunication module 220 may include, for example, the cellular module221, a Wi-Fi module 223, a Bluetooth module 225, a GNSS module 227, anNFC module 228, and an RF module 229. The cellular module 221 mayprovide a voice call, a video call, a text message service, or anInternet service through, for example, a communication network.According to an embodiment, the cellular module 221 may distinguish andauthenticate the electronic device 201 within a communication networkusing a SIM (e.g., the SIM card) 224. According to an embodiment, thecellular module 221 may perform at least some of the functions that theprocessor 210 may provide. According to an embodiment, the cellularmodule 221 may include a communication processor (CP). According to someembodiments, at least some (e.g., two or more elements) of the cellularmodule 221, the Wi-Fi module 223, the Bluetooth module 225, the GNSSmodule 227, and the NFC module 228 may be included within one integratedcircuit (IC) or an IC package. The RF module 229 may transmit/receive,for example, a communication signal (e.g., an RF signal). The RF module229 may include, for example, a transceiver, a power amp module (PAM), afrequency filter, a low noise amplifier (LNA), an antenna, and the like.According to another embodiment, at least one of the cellular module221, the Wi-Fi module 223, the Bluetooth module 225, the GNSS module227, and the NFC module 228 may transmit and receive RF signals througha separate RF module. The SIM 224 may include, for example, a cardcontaining a SIM and/or an embedded SIM, and may contain uniqueidentification information (e.g., an integrated circuit card identifier(ICCID)) or subscriber information (e.g., international mobilesubscriber identity (IMSI)).

The memory 230 (e.g., the memory 130) may include, for example, anembedded memory 232 or an external memory 234. The embedded memory 232may include at least one of, for example, a volatile memory (e.g., adynamic random access memory (DRAM), a static RAM (SRAM), a synchronousDRAM (SDRAM), and the like) and a non-volatile memory (e.g., a one-timeprogrammable read only memory (OTPROM), a PROM, an erasable andprogrammable ROM (EPROM), an electrically erasable and programmable ROM(EEPROM), a mask ROM, a flash ROM, a flash memory, a hard driver, or asolid state drive (SSD)). The external memory 234 may include a flashdrive, for example, a compact flash (CF), a secure digital (SD), amicro-SD, a mini-SD, an extreme digital (xD), a multi-media card (MMC),a memory stick, and the like. The external memory 234 may befunctionally and/or physically connected to the electronic device 201through various interfaces.

The sensor module 240 may, for example, measure a physical quantity ordetect an operation state of the electronic device 201, and may convertthe measured or detected information into an electric signal. The sensormodule 240 may include, for example, at least one of a gesture sensor240A, a gyro sensor 240B, an atmospheric pressure sensor 240C, amagnetic sensor 240D, an acceleration sensor 240E, a grip sensor 240F, aproximity sensor 240G a color sensor 240H (e.g., a red, green, blue(RGB) sensor), a biometric sensor 2401, a temperature/humidity sensor240J, an illuminance sensor 240K, and an ultraviolet (UV) sensor 240M.Additionally or alternatively, the sensor module 240 may include, forexample, an E-nose sensor, an electromyography (EMG) sensor, anelectroencephalogram (EEG) sensor, an electrocardiogram (ECG) sensor, aninfrared (IR) sensor, an iris sensor, and/or a fingerprint sensor. Thesensor module 240 may further include a control circuit for controllingone or more sensors included therein. According to some embodiments, theelectronic device 201 may further include a processor that is configuredto control the sensor module 240, as a part of the processor 210 orseparately from the processor 210, and may control the sensor module 240while the processor 210 is in a sleep state.

The input device 250 may include, for example, a touch panel 252, a(digital) pen sensor 254, a key 256, or an ultrasonic input device 258.The touch panel 252 may use at least one of, for example, a capacitivescheme, a resistive scheme, an infrared scheme, and an ultrasonicscheme. Further, the touch panel 252 may further include a controlcircuit. The touch panel 252 may further include a tactile layer andprovide a tactile reaction to a user. The (digital) pen sensor 254 mayinclude, for example, a recognition sheet, which is a part of the touchpanel or is separated from the touch panel. The key 256 may include, forexample, a physical button, an optical key or a keypad. The ultrasonicinput device 258 may detect an ultrasonic wave generated by an inputtool using a microphone (e.g., a microphone 288) of the electronicdevice 201, and may determine data corresponding to the detectedultrasonic wave.

The display 260 (e.g., the display 160) may include a panel 262, ahologram device 264, a projector 266, and/or a control circuit forcontrol thereof. The panel 262 may be embodied to be, for example,flexible, transparent, or wearable. The panel 262 and the touch panel252 may be embodied as one or more modules. According to an embodiment,the panel 262 may include a pressure sensor (or a force sensor) capableof measuring the intensity of the pressure with respect to a user'stouch. The pressure sensor may be integrated with the touch panel 252 ormay be implemented as one or more sensors separate from the touch panel252. The hologram 264 may show a three-dimensional image in the air byusing an interference of light. The projector 266 may display an imageby projecting light onto a screen. The screen may be located, forexample, inside or outside the electronic device 201. The interface 270may include, for example, an HDMI 272, a USB 274, an optical interface276, or a D-subminiature (D-sub) 278. The interface 270 may be includedin, for example, the communication interface 170 illustrated in FIG. 1 .Additionally or alternatively, the interface 270 may, for example,include a mobile high-definition link (MHL) interface, an SDcard/multi-media card (MMC) interface, or an infrared data association(IrDA) standard interface.

The audio module 280 may bilaterally convert, for example, a sound andan electric signal. At least some components of the audio module 280 maybe included in, for example, the input/output interface 150 illustratedin FIG. 1 . The audio module 280 may process sound information that isinput or output through, for example, a speaker 282, a receiver 284,earphones 286, the microphone 288, and the like. The camera module 291is a device that may photograph a still image and a dynamic image.According to an embodiment, the camera module 291 may include one ormore image sensors (e.g., a front sensor or a back sensor), a lens, animage signal processor (ISP), or a flash (e.g., LED or xenon lamp). Thepower management module 295 may manage, for example, power of theelectronic device 201. According to an embodiment, the power managementmodule 295 may include a power management IC (PMIC), a charger IC, or abattery or a fuel gauge. The PMIC may use a wired and/or wirelesscharging method. Examples of the wireless charging method may include,for example, a magnetic resonance method, a magnetic induction method,an electromagnetic wave method, and the like, and may further include anadditional circuit for wireless charging, for example, a coil loop, aresonance circuit, or a rectifier. The battery gauge may measure, forexample, a residual quantity of the battery 296, and a voltage, acurrent, or a temperature during charging. The battery 296 may include,for example, a rechargeable battery and/or a solar battery.

The indicator 297 may indicate a specific state (e.g., a booting state,a message state, a charging state, and the like) of the electronicdevice 201 or a part (e.g., the processor 210) of the electronic device201. The motor 298 may convert an electric signal into mechanicalvibrations, and may generate a vibration, a haptic effect, and the like.The electronic device 201 may include a mobile TV supporting device(e.g., GPU) which can process media data according to a certain standardsuch as digital multimedia broadcasting (DMB), digital videobroadcasting (DVB), mediaFLO™, and the like. Each of the above-describedcomponent elements according to the present specification may beconfigured with one or more components, and the names of thecorresponding component elements may vary based on the type ofelectronic device. According to various embodiments, some elements maybe omitted or other additional elements may be further included in theelectronic device (e.g., the electronic device 201). Alternatively, someof the components may be combined to form a single entity, and thus, mayequivalently execute functions of the corresponding elements prior tothe combination.

FIG. 3 is a block diagram illustrating a program module according tovarious embodiments. According to an embodiment, a program module 310(e.g., the program 140) may include an operating system (OS) forcontrolling resources related to an electronic device (e.g., theelectronic device 101) and/or various applications (e.g., theapplication programs 147) driven in the OS. The OS may include, forexample, Android™, Windows™, Symbian™, Tizen™, Bada™, or the like.Referring to FIG. 3 , the program module 310 may include a kernel 320(e.g., the kernel 141), middleware 330 (e.g., the middleware 143), anAPI 360 (e.g., the API 145), and/or applications 370 (e.g., theapplication programs 147). At least a portion of the program module 310may be preloaded on the electronic device, or may be downloaded from anexternal electronic device (e.g., the electronic device 102 or theelectronic device 104, or the server 106).

The kernel 320 may include, for example, a system resource manager 321and/or a device driver 323. The system resource manager 321 may control,assign, or collect system resources. According to an embodiment, thesystem resource manager 321 may include a process manager, a memorymanager, a file system manager, and the like. The device driver 323 mayinclude, for example, a display driver, a camera driver, a Bluetoothdriver, a shared memory driver, a USB driver, a keypad driver, a Wi-Fidriver, an audio driver, or an inter-process communication (IPC) driver.The middleware 330 may provide a function implemented by theapplications 370 in common or may provide various functions to theapplications 370 through the API 360 so that the applications 370 mayuse limited system resources within the electronic device. According toan embodiment, the middleware 330 may include, for example, at least oneof a runtime library 335, an application manager 341, a window manager342, a multimedia manager 343, a resource manager 344, a power manager345, a database manager 346, a package manager 347, a connectivitymanager 348, a notification manager 349, a location manager 350, agraphic manager 351, and a security manager 352.

The runtime library 335 may include, for example, a library module thata compiler uses in order to add new functions through a programminglanguage while the applications 370 are executed. The runtime library335 may perform input/output management, memory management, orarithmetic function processing. The application manager 341 may, forexample, manage the life cycle of at least one of the applications 370.The window manager 342 may manage graphical user interface (GUI)resources used on a screen. The multimedia manager 343 may identifyformats that are utilized for the reproduction of various media filesand encode or decode a media file using a codec suitable for acorresponding format. The resource manager 344 may manage the sourcecode of the applications 370 or the space of a memory. The power manager345 may, for example, manage the capacity or power of a battery and mayprovide power information utilized for the operation of the electronicdevice. According to an embodiment, the power manager 345 may operatetogether with a basic input/output system (BIOS). The database manager346 may generate, search for, or change a database to be used, forexample, in the applications 370. The package manager 347 may manage theinstallation or updating of an application distributed in the form of apackage file.

The connectivity manager 348 may manage, for example, a wirelessconnection. The notification manager 349 may provide events such as anarrival message, an appointment, proximity notification, and the like toa user. The location manager 350 may manage, for example, locationinformation of the electronic device. The graphic manager 351 maymanage, for example, a graphic effect to be provided to a user and auser interface relating to the graphic effect. The security manager 352may provide, for example, system security or user authentication.According to an embodiment, the middleware 330 may further include atelephony manager for managing a voice call function or a video callfunction of the electronic device, or a middleware module that formscombinations of the functions of the above-described components.According to an embodiment, the middleware 330 may provide a modulespecialized for each type of OS. Furthermore, the middleware 330 maydynamically remove some of the existing components, or may add newcomponents. The API 360 is, for example, a set of API programmingfunctions, and may be provided with a different configuration accordingto an OS. For example, in the case of Android or iOS, one API set may beprovided for each platform, and in the case of Tizen, two or more APIsets may be provided for each platform.

The applications 370 may include, for example, a home application 371, adialer application 372, a short message service (SMS)/multimedia messageservice (MMS) application 373, an instant message (IM) application 374,a browser application 375, a camera application 376, an alarmapplication 377, contacts application 378, a voice dial application 379,an e-mail application 380, a calendar application 381, a media playerapplication 382, an album application 383, a watch application 384, ahealth care application (e.g., measure exercise quantity or bloodsugar), and environment information (e.g., atmospheric pressure,humidity, temperature information, and the like). According to anembodiment, the applications 370 may include an information exchangeapplication that supports exchanging information between the electronicdevice and an external electronic device. The information exchangeapplication may include, for example, a notification relay applicationfor notifying an external electronic device of certain information or adevice management application for managing an external electronicdevice. For example, the notification relay application may transfer, toan external electronic device, notification information generated fromother applications of the electronic device, or may receive notificationinformation from an external electronic device and provide the receivednotification information to a user. The device management applicationmay install, delete, or update the function of an external electronicdevice communicating with the electronic device (e.g., a function ofturning on/off the external electronic device itself (or somecomponents) or a function of adjusting luminance (or a resolution) ofthe display), applications operating in the external electronic device.According to an embodiment, the applications 370 may include anapplication (e.g., a health care application of a mobile medical device)designated according to an attribute of the external electronic device.According to an embodiment, the applications 370 may include anapplication received from the external electronic device. At least aportion of the program module 310 may be implemented (e.g., executed) insoftware, firmware, hardware (e.g., the processor 210), or a combinationof two or more thereof, and may include, for example, a module, aprogram, a routine, a set of instructions, and/or a process forperforming one or more functions.

According to various embodiments, an electronic device (e.g., theelectronic device 101) may include a housing; a touch screen display(e.g., the display 160) that is exposed through a portion of thehousing; a wireless communication circuit (e.g., the communicationinterface 170); a microphone (e.g., the microphone 288); a speaker(e.g., the speaker 282); at least one processor (e.g., the processor120) that is located inside the housing and is electrically connected tothe touch screen display, the wireless communication circuit, themicrophone, and the speaker; and a memory (e.g., the memory 130) that islocated inside the housing and is electrically coupled to the at leastone processor, such that the memory is configured to at leasttemporarily store a plurality of reference sequences of states, each ofthe plurality of reference sequences of states is associated withindividual operations of the electronic device or individualcombinations of the operations, and when the memory is performed, the atleast one processor receives a first sequence of touch inputs from auser via the touch screen display; sequentially changes the states ofthe electronic device to perform operations of the electronic device orthe combinations of the operations in response to the first sequence ofthe touch inputs; stores a first sequence of the changed states in thememory; compares the first sequence of the changed states with at leasta portion of at least one of the plurality of reference sequences ofstates; and stores instructions for causing at least one result from thecomparison to be stored in the memory.

According to various embodiments, the instructions may cause the atleast one processor to receive at least one touch input less than thefirst sequence of the touch inputs via the touch screen display afterthe at least one result is stored, to determine that the at least onetouch input is associated with the at least one result and/or the atleast one of the plurality of reference sequences of states, and toprovide an indication via the touch screen display and/or the speaker,and the indication may indicate that at least a portion of the firstsequence of the changed states is repeated using the indication withoutproviding the first sequence of the touch inputs by the user.

According to various embodiments, the instructions may cause the atleast one processor to collect first context information indicating aninternal operation of the electronic device or an external context whenthe states of the electronic device are sequentially changed, to storethe context information in the memory to correspond to the firstsequence of the changed states, to collect second context informationindicating the internal operation of the electronic device or theexternal context at the time of the at least one touch input, and toprovide the indication via the touch screen display and/or the speakerwhen the first context information and the second context informationcorrespond to each other.

According to various embodiments, the instructions may cause the atleast one processor to update the number of executions of the changedstates of the first sequence to store the updated number of executionsin the memory, and to provide the indication via the touch screendisplay and/or the speaker when the updated number of executionscorresponds to a designated condition.

According to various embodiments, the instructions may cause the atleast one processor to provide the indication via the touch screendisplay and/or the speaker when a time point when the operation or theoperations is applied as functions of the electronic device correspondsto the designated condition.

According to various embodiments, the instructions may cause the atleast one processor to begin to sequentially change the states beforethe first sequence of the touch inputs is completely received.

According to various embodiments, the instructions may cause the atleast one processor to receive a voice input via the microphone afterthe at least one result is stored, to determine that the voice input isassociated with the at least one result and/or the at least one of theplurality of reference sequences of states, and to provide an indicationvia the touch screen display and/or the speaker, and the indication mayindicate that at least a portion of the first sequence of the changedstates is repeated using the indication without providing the firstsequence of the touch inputs by the user.

According to various embodiments, the instructions may cause the atleast one processor to provide an indication that suggests designating amacro corresponding to the at least one result and to designate themacro corresponding to the at least one result in accordance with thevoice input via the microphone or the touch input via the touch screendisplay for the indication that suggests designating the macro, and theindication provided via the touch screen display and/or the speakerindicates the macro that is designated to correspond to the at least oneresult.

According to various embodiments, the instructions may cause the atleast one processor to determine a priority of at least one indicationto provide based on the designated condition of the electronic deviceand to provide the at least one indication via the touch screen displayand/or the speaker based on the determined priority, and the indicationmay indicate that at least a portion of the first sequence of thechanged states is repeated using the indication without providing thefirst sequence of the touch inputs.

According to various embodiments, the instructions may cause the atleast one processor to determine an application domain that performs theoperations of the electronic device or the combinations of theoperations based on the number of times the application domain is used,at the time of a voice input via the microphone or the touch input viathe touch screen display for performing the operations of the electronicdevice or the combinations of the operations, to generate at least oneindication including information of the application domain, and toprovide the at least one indication via the touch screen display and/orthe speaker, and the indication may indicate that at least a portion ofthe first sequence of the changed states is repeated in the applicationdomain by using the indication without providing the first sequence ofthe touch inputs.

According to various embodiments, an electronic system may include: atouch screen display (e.g., the display 160); at least one communicationcircuit (e.g., the communication interface 170); a microphone (e.g., themicrophone 288); a speaker (e.g., the speaker 282); one or moreprocessors (e.g., the processor 120) that is operably connected to thetouch screen display, the communication circuit, the microphone, and thespeaker; and at least one memory (e.g., the memory 130) that is operablyconnected to the one or more processors, such that the memory isconfigured to at least temporarily store a plurality of referencesequences of states, and each of the plurality of reference sequences ofstates is associated with individual operations of an electronic deviceor individual combinations of the operations, and when the memory isperformed, the one or more processors acquires a first sequence of touchinputs from a user via the touch screen display, sequentially changesthe states of the electronic system to perform the operations of theelectronic system or the combinations of the operations in response tothe first sequence of the touch inputs, stores a first sequence of thechanged states in the memory, compares the first sequence of the changedstates with at least a portion of at least one of the plurality ofreference sequences of states, and stores instructions for causing atleast one result from the comparison to be stored in the memory.

According to various embodiments, the electronic system may furtherinclude: a mobile device that includes a first processor; and a serverthat includes a second processor, such that the instructions cause thefirst processor to acquire the first sequence of the touch inputs viathe touch screen display and the second processor to sequentially changethe states of the electronic device to perform the operations of theelectronic device or the combinations of the operations in response tothe first sequence of the touch inputs.

According to various embodiments, the instructions may cause the secondprocessor to store the first sequence of the changed states in thememory, to compare the first sequence of the changed states with the atleast a portion of at least one of the plurality of reference sequencesof states, and to store the at least one result from the comparison inthe memory.

According to various embodiments, the instructions may cause the one ormore processors to receive at least one touch input less than the firstsequence of the touch inputs via the touch screen display after the atleast one result is stored, to determine that the at least one touchinput is associated with the at least one result and/or the at least oneof the plurality of reference sequences of states, to provide anindication via the touch screen display and/or the speaker, to receive aselection of the indication via the touch screen display or the speaker,and to at least partially repeat at least a portion of the firstsequence of the changed states in at least partially response to theselection.

According to various embodiments, the instructions may cause the one ormore processors to begin to sequentially change the states before thefirst sequence of the touch inputs is completely received.

According to various embodiments, an electronic device may include: ahousing; a touch screen display (e.g., the display 160) that is exposedthrough a portion of the housing; a wireless communication circuit(e.g., the communication interface 170); a microphone (e.g., themicrophone 288); a speaker (e.g., the speaker 282); at least oneprocessor (e.g., the processor 120) that is located inside the housingand is electrically connected to the touch screen display, the wirelesscommunication circuit, the microphone, and the speaker; and a memory(e.g., the memory 130) that is located inside the housing and iselectrically connected to the at least one processor, such that thememory is configured to at least temporarily store a plurality ofreference sequences of states and the plurality of reference sequencesof states is associated with individual operations of the electronicdevice or individual combinations of the operations, and when the memoryis performed, the at least one processor receives a first sequence ofinputs from a user, sequentially changes the states of the electronicdevice to perform the operations of the electronic device or thecombinations of the operations in response to the first sequence of theinputs, stores a first sequence of the changed states in the memory,compares the first sequence of the changed states with at least aportion of at least one of the plurality of reference sequences ofstates, and stores instructions for causing at least one result from thecomparison to be stored in the memory.

According to various embodiments, an electronic device may include: ahousing; a touch screen display (e.g., the display 160) that is exposedthrough a portion of the housing; a wireless communication circuit(e.g., the communication interface 170); a microphone (e.g., themicrophone 288); a speaker (e.g., the speaker 282); at least oneprocessor (e.g., the processor 120) that is located inside the housingand is electrically connected to the touch screen display, the wirelesscommunication circuit, the microphone, and the speaker; and a memory(e.g., the memory 130) that is located inside the housing and iselectrically connected to the at least one processor, such that thememory is configured to at least temporarily store a plurality ofreference sequences of states and the plurality of reference sequencesof states is associated with individual operations of the electronicdevice or individual combinations of the operations, and when the memoryis performed, the at least one processor receives one or more touchinputs from a user via the touch screen display, sequentially changesthe states of the electronic device to perform the operations of theelectronic device or the combinations of the operations in response tothe one or more touch inputs, stores a first sequence of the changedstates in the memory, compares the first sequence of the changed stateswith at least a portion of at least one of the plurality of referencesequences of states, and stores instructions for causing at least oneresult from the comparison to be stored in the memory.

According to various embodiments, an electronic device (e.g., theelectronic device 101) may include: a housing; a touch screen display(e.g., the display 160) that is exposed through a portion of thehousing; an input device (e.g., the input/output interface 150); atleast one processor (e.g., the processor 120) that is located inside thehousing and is electrically connected to the touch screen display (e.g.,the display 160) and the input device; and a memory (e.g., the memory130) that is located inside the housing and is electrically connected tothe at least one processor, such that when the memory is performed, theat least one processor receives a first sequence of inputs from a uservia the input device, sequentially changes a first sequence of states ofthe electronic device to perform a first operation of the electronicdevice in response to the first sequence of the inputs, storesinformation associated with the first sequence of the states in thememory, provides an indication via the touch screen display at leastbased on the information associated with the first sequence of thestates stored in the memory, the indication indicating availability ofthe first operation, and stores, when the indication is selected,instructions causing performance of the first operation without asequential change of the first sequence of the states.

According to various embodiments, the instructions may cause the atleast one processor to receive at least one input corresponding to thefirst sequence of the inputs and to perform, when the indication isselected, the first operation by a change in the at least one state ofthe electronic device, and a first number of the at least one state isless than a second number of the first sequence of the states.

According to various embodiments, the memory may be configured to atleast temporarily store a plurality of reference sequences of states andthe plurality of reference sequences of states may be associated withindividual operations of the electronic device or individualcombinations of the operations, and the instructions may cause the atleast one processor to compare the first sequence of the states with atleast a portion of at least one of the plurality of reference sequencesof states and to store at least one result of the comparison in thememory as the information associated with the first sequence.

According to various embodiments, the instructions may cause the atleast one processor to provide an indication that suggests designating amacro corresponding to the first sequence of the states based on theinformation associated with the first sequence, to designate the macrocorresponding to the first sequence based on an input for the indicationthat suggests designating the macro via the input device, and to providethe indication corresponding to the macro via the touch screen display.

According to various embodiments, the instructions may cause the atleast one processor to collect, when the first sequence of the states issequentially changed, first context information indicating an internaloperation of the electronic device or an external context to store thecollected first context information in the memory, and to determine theindication in accordance with a comparison result between second contextinformation indicating the internal operation of the electronic deviceor the external context collected at the time of execution of theelectronic device or execution of an application thereof and the firstcontext information.

According to various embodiments, the instructions may cause the atleast one processor to determine the indication depending on the numberof times the first operation stored in the memory is performed.

According to various embodiments, the instructions may cause the atleast one processor to determine the indication depending on a timepoint when the first operation is applied as a function of theelectronic device.

According to various embodiments, in a machine-readable storage mediumin which a program for executing a method of generating a hint forexecuting an operation of an electronic device (e.g., the electronicdevice 101) is recorded, the method may include: receiving a firstsequence of inputs from a user; sequentially changing a first sequenceof states of the electronic device to perform a first operation of theelectronic device in response to the first sequence of the inputs;storing information associated with the first sequence of the states;providing an indication via a touch screen display of the electronicdevice at least based on the stored information associated with thefirst sequence of the states, such that the indication indicatesavailability of the first operation; and performing the first operationwithout a sequential change of the first sequence of the states when theindication is selected.

FIG. 4 is a flowchart illustrating a hint generation operation forexecuting an electronic device (e.g., the electronic device 101)according to various embodiments. Referring to FIG. 4 , the electronicdevice may generate and provide a hint (also referred to as anindication) to a user in a processor (e.g., the processor 120) of theelectronic device based on a usage pattern of the user. For example, thehint may be a text that informs the user of a voice to be input so thatan operation of the electronic device can be executed through a voiceinput. For example, the operation of the electronic device may be anoperation of the electronic device that is expected to be executed bythe user.

In operation 410, the electronic device (e.g., the processor 120) maycollect logs of the electronic device.

According to an embodiment, the electronic device may collect logs ofthe electronic device during the operation (e.g., when the user uses theelectronic device). For example, when executing at least one operationaccording to a user input, the electronic device may collect the logsrelated to the execution of the at least one operation. For example, theelectronic device may any collect screen transformation information thatis displayed when the user uses the electronic device.

That is, according to an embodiment, the logs may include recording ofscreens which are sequentially displayed by the electronic device inorder for the electronic device to execute a function including the atleast one operation according to the user input and/or recording ofoperations which are sequentially executed by the electronic device (andthus “screen transformation information”). For example, a single screendisplayed according to one operation executed by the electronic deviceor the execution of the one operation may be referred to as a state, andrecording of the operations sequentially executed by the electronicdevice or recording of screens sequentially displayed to correspond tothe sequentially executed operations may be referred to as a statesequence log (hereinafter, also referred to as a sequence of states).For example, the state sequence log may be obtained by sequentiallyarranging the operations executed according to the user input.

According to an embodiment, the logs may further include information onthe user input. For example, the user input may be various, such as avoice input using a microphone, a touch input on a touch screen, a usermanipulation on buttons (or keys), and/or an input using a remotecontroller.

According to an embodiment, the logs may further include contextinformation (also referred to as context) when the electronic devicereceives, from a user, a user input that causes execution of a specificfunction of the electronic device. For example, the context informationmay indicate an internal operation of the electronic device or anexternal situation, and may be various, such as a location, a date, atime, and/or schedule information stored in the electronic device.

In operation 420, the electronic device (e.g., the processor 120) mayanalyze collected logs and may detect a path rule indicatingsequentially changed states of the electronic device. For example, thepath rule may be obtained by the electronic device dividing the functionof an application into unit operations.

According to an embodiment, the electronic device may select statescorresponding to the use intention or purpose of the electronic devicein such a manner that the collected logs are wholly or partially matchedwith a previously stored state sequence log (also referred to as areference sequence of states), and may find the path rule allowing theselected states to be sequentially executed. For example, the previouslystored state sequence log may be updated according to the operation ofthe electronic device depending on the user input. For example,according to the detection of the path rule, the electronic device mayconfirm which function of the electronic device is used, in whichcontext the function is used, and/or through which input means thefunction is used.

According to another embodiment, the electronic device may analyze astate log that is repeatedly used through pattern analysis of thecollected state logs to thereby extract a new path rule. For example,when the state log is performed a designated reference number of timesor more, the electronic device may confirm that a user has repeatedlyperformed the above-described function through analysis of the collectedstate logs, and may newly acquire a path rule including the statesequence.

The above-described path rule will be described later with reference toFIG. 28 .

In operation 430, the electronic device (e.g., the processor 120) maydetermine a priority of the path rule and may select the path rule basedon the priority.

According to an embodiment, the electronic device may determine and/orchange the priority of the path rule corresponding to a hint to beprovided to a user based on information included in the log. Forexample, the information included in the log may include information ona user input and/or context information.

According to an embodiment, the priority of the path rule may bedetermined so that a hint converted into a natural languagecorresponding to the path rule can be provided to the user in variousways according to a user occasion. For example, the electronic devicemay determine the user occasion based on the context information and/ora frequency of use of the path rule.

According to an embodiment, when a plurality of path rules is detectedthrough log analysis, the electronic device may determine the priorityof the plurality of path rules in various ways depending on the purposein which the plurality of path rules provides a hint to the user. Forexample, when the hint can be provided so that the user can easilyprocess the path rule, which is frequently used via a touch input, inthe natural language by the touch input, a high priority may be assignedto the path rule which is used via the touch input but is not utilizedwell via the voice input. This can increase the utilization of the hintby reducing interface steps of a frequently used function. By way ofanother example, assuming that a user uses the same path rule mainly viathe voice input at home and controls the same path rule via the touchinput at the office, when a user's location is the office at the time ofproviding a hint to the user, the electronic device may exclude the pathrule of the hint that suggests the voice input to the user.

In operation 440, the electronic device (e.g., the processor 120) mayconvert the path rule selected based on the priority into a naturallanguage hint.

According to an embodiment, the electronic device may convert the pathrule selected based on the priority into a hint in the form of thenatural language through a natural language generation (NLG) techniqueor mapping with standard utterance matched with the path rule.

In operation 450, the electronic device (e.g., the processor 120) mayprovide the hint converted into the natural language.

According to an embodiment, the electronic device may automaticallyprovide the hint when the electronic device is activated or when theapplication of the electronic device is executed.

According to another embodiment, the electronic device may provide thehint according to a user input such as a touch input, a voice input, auser manipulation on hardware buttons (or keys), and/or an input using aremote controller. According to an embodiment, the electronic device mayprovide the hint converted into the natural language through a display.For example, when the hint in the form of the converted natural languageis “Play the latest music”, “call Amy”, or the like according to theabove-described operations, the electronic device may display the hintsuch as “Play the latest music”, “call Amy”, or the like on apredetermined region 510 of the display as shown in FIG. 5 .

According to another embodiment, the electronic device may output thehint converted into the natural language as voice through a speakerusing a text to speech (TTS) technique.

According to still another embodiment, the electronic device may providethe hint converted into the natural language through the display and thespeaker.

According to an embodiment, when a user utters or touches the hintdisplayed on the display, the electronic device may sequentially andautomatically execute the states of the path rule corresponding to thehint. For example, the electronic device may display a final state inwhich the states are sequentially executed on the touch screen. By wayof another example, the electronic device may sequentially display allthe sequential executions of the states on the touch screen.

According to an embodiment, the electronic device may automaticallyexecute the above-described operations of FIG. 4 upon the execution ofthe electronic device or the execution of the application of theelectronic device. Alternatively, the electronic device may execute theabove-described operations of FIG. 4 according to a user input such as atouch input, a voice input, a user manipulation on buttons (or keys),and/or an input using a remote controller.

FIG. 6 is a flowchart illustrating a hint generation operation forexecuting the electronic device (e.g., the processor of the server 106)according to various embodiments. Referring to FIG. 6 , the electronicdevice may receive a log from an external electronic device, maygenerate a hint of the external electronic device based on a usagepattern of a user of the external electronic device, and may transmitthe generated hint to the external electronic device.

In operation 610, the electronic device (e.g., the processor of theserver 106) may receive a log of the external electronic device from theexternal electronic device via a wireless communication circuit.

According to an embodiment, the log may include a recording of screenswhich were sequentially displayed by the external electronic device, inwhich the external electronic device executed a function including atleast one operation according to user input, and/or recording ofoperations which were sequentially executed by the external electronicdevice. For example, a single screen displayed according to oneoperation executed by the external electronic device or execution of theone operation may be referred to as a state, and recording of operationssequentially executed by the external electronic device or recording ofscreens sequentially displayed to correspond to the sequentiallyexecuted operations may be referred to as a state sequence log. Forexample, the state sequence log may be obtained by sequentiallyarranging the operations executed according to the user input.

According to an embodiment, the log may further include information onthe user input of the external electronic device. For example, the userinput of the external electronic device may be various, such as a voiceinput using a microphone, a touch input on a touch screen, a usermanipulation on buttons (or keys), and/or an input using a remotecontroller.

According to an embodiment, the log may further include contextinformation (also referred to as context) when the external electronicdevice receives a user input that causes execution of a specificfunction of the external electronic device, from a user. For example,the context information may be various, such as a location, a date, atime, and/or schedule information stored in the electronic device.

In operation 620, the electronic device (e.g., the processor of theserver 106) may analyze the received log and may detect a path rule ofthe external electronic device, the path rule thereby indicating thesequential change of operation states of the external electronic device.For example, the path rule may be obtained by the electronic devicedividing the function of an application into unit operations.

According to an embodiment, the electronic device may predict a usageintention or user-defined purpose of the external electronic device, mayselect states corresponding to the use intention or purpose of theexternal electronic device in such a manner that the received log iswholly or partially matched with a previously stored state sequence log,and may find a path rule allowing the selected states to be sequentiallyexecuted. For example, the previously stored state sequence log may beupdated according to the log of the external electronic device. Forexample, according to the detection of the path rule, the electronicdevice may confirm which function of the external electronic device isused, in which context the function is used, and/or through which inputmeans the function is used.

According to another embodiment, the electronic device may analyze astate log that is repeatedly used through pattern analysis of thereceived state logs of the external electronic device to thereby extracta new path rule. For example, when the state log is performed adesignated reference number of times or more, the electronic device mayconfirm that a user of the external electronic device has repeatedlyperformed the above-described function through analysis of the receivedstate logs of the external electronic device, and may newly acquire apath rule including the state sequence.

In operation 630, the electronic device (e.g., the processor of theserver 106) may determine a priority of the path rule of the externalelectronic device, and may select the path rule of the externalelectronic device based on the determined priority.

According to an embodiment, the electronic device may set and/or changethe priority of the path rule to correspond to a hint to be provided tothe user of the external electronic device, based on informationincluded in the log of the external electronic device.

According to an embodiment, the priority of the path rule may bedetermined so that a hint converted into a natural languagecorresponding to the path rule of the external electronic device can beprovided to the user in various ways according to a user occasion of theexternal electronic device. For example, the electronic device maydetermine the user occasion of the external electronic device based onthe context information of the external electronic device and/or afrequency of use of the path rule of the external electronic device.

According to an embodiment, when a plurality of path rules of theexternal electronic device is detected through log analysis, theelectronic device may determine the priority of the plurality of pathrules in various ways depending on the purpose in which the plurality ofpath rules provides the hint to the user of the external electronicdevice.

In operation 640, the electronic device (e.g., the processor of theserver 106) may convert the path rule of the external electronic deviceselected based on the priority into a natural language hint.

According to an embodiment, the electronic device may convert the pathrule of the external electronic device selected based on the priorityinto a hint in the form of the natural language through a naturallanguage generation technique or mapping with standard utterance matchedwith the path rule.

In operation 650, the electronic device (e.g., the processor of theserver 106) may transmit the hint (e.g., as converted into the naturallanguage) to the external electronic device via a wireless communicationcircuit.

According to an embodiment, when receiving a request to provide the hintfrom the external electronic device, the electronic device may transmitthe hint to the external electronic device. For example, the externalelectronic device may automatically make the request to provide the hintfrom the electronic device when the electronic device is performed orwhen the application of the electronic device is executed. By way ofanother example, the external electronic device may make the request toprovide the hint from the electronic device according to a user inputsuch as a touch input, a voice input, a user manipulation on hardwarebuttons (or keys), and/or an input using a remote controller.

FIG. 7 is a flowchart illustrating the operation of the electronicdevice (e.g., the electronic device 101) that generates a path rulethrough user log collection of the electronic device according tovarious embodiments. Referring to FIG. 7 , the electronic device maygenerate and/or determine a path rule for generating a hint to beprovided to a user.

In operation 710, the electronic device (e.g., the electronic device101) may receive a user input.

According to an embodiment, the user input may be various, such as avoice input, a touch input, a user manipulation on buttons (or keys),and/or an input using a remote controller.

According to an embodiment, the user input may be a single input thatcauses the electronic device to perform one operation, or a plurality ofconsecutive inputs that causes the electronic device to performoperations consecutively.

According to an embodiment, when the user input includes at least onevoice input via a microphone, the electronic device (e.g., the processor120) may collect utterance path rules of the electronic device based onthe voice input in operation 720. For example, the electronic device maydetermine which utterance path rule is input as a voice in order for theelectronic device to perform specific operations sequentially, usingnatural language understanding (NLU). For example, the utterance pathrule may be log data for sequential operations of the electronic device.

In operation 730, the electronic device (e.g., the processor 120) maystore the collected utterance path rules in an utterance path ruletable. For example, the utterance path rule table may be stored in amemory of the electronic device.

According to an embodiment, when the user input includes at least onetouch input on the touch screen, the electronic device (e.g., theprocessor 120) may generate a touch state log based on the touch inputin operation 740. For example, the electronic device may generate thetouch state log obtained by sequentially combining operations performedto correspond to each of the touch inputs which are consecutivelyreceived. For example, the electronic device may generate the touchstate log obtained by sequentially combining the states of theelectronic device which are changed by the touch input.

In operation 750, the electronic device (e.g., the processor 120) maydetermine a touch path rule based on the touch state log. For example,the electronic device may confirm to which portion a portioncorresponding to the state sequence log in which the touch state log isstored corresponds, by comparing the touch state log with a statesequence log stored in an existing rule database (or the memory of theelectronic device) through a rule mapper process, and may extract thetouch path rule corresponding to the touch state log from the storedstate sequence log.

The rule mapper process is a process of finding a previously definedpath rule by analyzing the collected logs (e.g., the touch state logs),and will be described in detail with reference to FIG. 8 .

FIGS. 8A to 8D are diagrams illustrating an operation of determining apath rule of the electronic device (e.g., the electronic device 101)depending on a user input according to various embodiments.

According to an embodiment, the path rule may be stored in the form of atable that defines a path rule including a representative dialog for afinal result that can be achieved by a combination of various operationsfor performing a specific function of the electronic device andincluding information of a sequence of states for sequentially executingthe various operations. For example, the representative dialog mayinclude user intention for performing the specific function.

For example, the path rule may comply with a hierarchy structure of auser interface (UI) through which a user uses a specific function of theelectronic device, and the hierarchy structure may be expressed in astate structure according to a screen and/or a function. The path rulemay be obtained by defining a state sequence for a path that can beprocessed in the forward direction in the hierarchical structure of theUI.

For example, as shown in FIG. 8A, the path rule may be defined in theform of a table that includes a representative dialog 801 for the finalresult that can be achieved by the combination of the various operations(functions) for performing the function of the electronic device, a pathrule ID 803 for identifying the path rule, a path rule (name) 805indicating operations sequentially executed in order for the electronicdevice to achieve the final result, and a path rule (state) 807indicating a combination of state IDs corresponding to each of theoperations.

According to an embodiment, the electronic device may assign the stateID for each of the operations of the electronic device. For example, theelectronic device may assign the state ID for each operation ofapplications in advance. Referring to FIG. 8B, a gallery application mayinclude execution, photo view, photo search, photo selection, selectedphoto removal, album view, album selection, execution of animationfunction to apply an animation effect to photos included in the selectedalbum, and execution of collage function to combine the photos includedin the selected album of the gallery application, or may includeexecution of shutdown function to shut down the gallery application. Inaddition, for example, a state ID of State_1 may be assigned to theexecution of the gallery application, a state ID of State_2 may beassigned to the photo view, a state ID of State_3 may be assigned to thephoto search, a state ID of State_4 may be assigned to the photoselection, a state ID of State_5 may be assigned to the selected photoremoval, a state ID of State_6 may be assigned to the album view, astate ID of State_7 may be assigned to the album selection, a state IDof State_8 may be assigned to the execution of animation function, astate ID of State_9 may be assigned to the execution of collagefunction, and a state ID of State_10 may be assigned to the execution ofshutdown function.

According to an embodiment, the electronic device may be storing thestate sequence log that is sequential operation recording of theelectronic device which is pre-set or generated according to a previoususer input. In addition, the electronic device may store user inputinformation for allowing each operation to be performed and applicationinformation in which each operation is performed.

According to another embodiment, as shown in FIG. 8C, the electronicdevice may store the state sequence log as a state ID corresponding toeach operation. Referring to FIG. 8C, the electronic device mayassociate, with corresponding user inputs 811, the state IDs 813corresponding to respective operations which have been sequentiallyexecuted according to the sequential user inputs 811, and may store theresult. In addition, the electronic device may associate evenapplication information 815 through which the operations are executedwith the state IDs 813, and may store the result. For example, theelectronic device may associate, with the application information 815corresponding to the respective operations such as “ . . . Gallery,Gallery, Gallery, Gallery, Gallery, Gallery, Gallery, Gallery, Gallery,Gallery, Gallery . . . ”, the sequential user inputs 811 such as “ . . .touch, touch, touch, touch, keyboard, touch, voice, touch, touch, touch,touch . . . ” and the state IDs 813 such as “ . . . State_2, State_10,State_1, State_2, State_3, State_4, State_5, State_6, State_7, State_2,State 4, . . . ” which are combinations of the state IDs correspondingto the respective operations executed according to the user inputs 811.

According to an embodiment, when the state of an application for a userinput (e.g., a multi-input) is changed, the state sequence logs may becollected, and may be collected together with the name of theapplication, the type of the user input, a time context, and the like aswell as the state ID. The electronic device may find a pattern matchedwith the user input from the state sequence log in various ways usingthe collected state sequence logs to generate a path rule.

According to an embodiment, the electronic device may determine a pathrule for processing a dialog of a user based on the previously storedstate sequence log described above.

For example, referring to FIG. 8A, in order to process a first dialog of“delete photo taken yesterday”, the electronic device may expect toselect this gallery application. For example, the electronic device maypredict operations in the order according to a path rule of ‘Appexecution—photo view—search—photo selection—deletion’ in order toprocess the first dialog. For example, the electronic device may find,from the previously stored state sequence log of the gallery applicationof the electronic device, ‘App execution—photo view—search—photoselection—deletion’ that is an operation path rule corresponding toPath_Rule_1 that is a path rule ID of the first dialog and/or a partmatched with the path rule indicating combinations of the state IDscorresponding to the operation path rule, and may generate the path rulecorresponding to the first dialog. For example, the electronic devicemay find a part 821 completely matched with the path rule of the firstdialog from the state sequence log as shown in FIG. 8D, and may generatethe found part 821 as a path rule corresponding to the first dialog.

For example, referring to FIG. 8A, the electronic device may predictselecting the gallery application in order to process a second dialog of‘Show my Hawaii photos as album animation’. For example, the electronicdevice may predict operations in the order according to a path rule of‘App execution—album view—album selection—animation’ in order to processthe second dialog. For example, the electronic device may find, from thepreviously stored state sequence log of the gallery application of theelectronic device, ‘App execution—album view—album selection—animation’that is an operation path rule corresponding to Path_Rule_2 that is anID of the second dialog and/or a part matched with the path ruleindicating combinations of the state IDs corresponding to the operationpath rule, and may generate a path rule corresponding to the seconddialog. For example, when there is no part completely matched with thepath rule of the second dialog from the previously stored state sequencelog, the electronic device may find a part 823 partially matchedtherewith and may generate the path rule corresponding to the seconddialog.

According to another embodiment, when it is assumed that transmission ofa new message to an acquaintance by a user through a short messageservice (SMS) is performed by the electronic device via a touch input,according to the above-described embodiment, the electronic device maygenerate a log for an operation state of a message application, and theuser may extract a path rule corresponding to a dialog of “transmissionof new message by user through SMS” through a rule mapper processor andmay store the corresponding path rule in a touch path rule.

In operation 760, the electronic device (e.g., the processor 120) maystore the touch path rule in a touch path rule table. For example, thetouch path rule table may be stored in the memory of the electronicdevice.

In operation 770, the electronic device (e.g., the processor 120) maycombine an utterance path rule of an utterance path rule table and thetouch path rule of the touch path rule table and may store the combinedresult in an utterance/touch combined path rule table. For example, theutterance/touch combined path rule table may be stored in the memory ofthe electronic device. For example, when the electronic device executesa first operation according to a first touch input, executes a secondoperation according to a second touch input, executes a third operationaccording to a first voice input, and executes a fourth operationaccording to a third touch input, the electronic device may store thefirst operation—the second operation—the third operation—the fourthoperation as a single path rule. For example, the electronic device maymake a database together with the number of times each path rule isperformed or the like, and may store the database. For example, theelectronic device may also store information indicating whether eachpath rule has been performed through a touch or a voice input so as tobe distinguished.

According to an embodiment, when a plurality of path rules is generatedthrough the above-described operations or when a plurality of path rulesis generated according to above-described operations 710 to 770repeatedly performed, the electronic device (e.g., the processor 120)may assign a priority for providing a hint to each path rule included inthe utterance/touch combined path rule table according to a designatedcondition, and may store the result in a hint path rule table inoperation 780. For example, the electronic device may give a hint to auser by assigning a priority to a path rule that the user usesfrequently in an existing touch method but does not use via voiceutterance, in order to expand user experience to the voice utterance.For example, the electronic device executes a gallery application toperform an operation of collecting photos in a specific place accordingto a user input into a single album. Here, the user input every time theoperation is performed may be a touch input rather than a voice input.In such a case, the electronic device may provide a hint indicating thatthe operation can be performed via the voice utterance, which can help auser to easily operate a function used through a voice command.

Meanwhile, in the above-described embodiment of FIG. 7 , it has beendescribed that the electronic device generates the path rule throughuser log collection of electronic device. However, according to anotherembodiment, the electronic device (e.g., the server 106) may receiveoperations executed by an external electronic device according to a userinput or recording of the operations from the external electronic devicevia a wireless communication circuit, and may be operated in a methodsimilar to the above-described operation (corresponding method). Forexample, the electronic device may execute operations of collectingutterance path rules of the external electronic device, storing theutterance path rules in the utterance path rule table, generating thetouch state log of the external electronic device, determining the touchpath rule based on the touch state log, storing the touch path rule inthe touch path rule table, combining the utterance path rule and thetouch path rule to store the combined result in the utterance/touchcombined path rule table, and assigning a priority for providing a hintto each path rule included in the utterance/touch combined path ruletable to store the assigned result in the hint path rule table. Forexample, the electronic device may generate the path rule through logcollection for each of a plurality of external devices (users), therebystoring the hint path rule table for each of the plurality of externaldevices (users).

FIG. 9 is a flowchart illustrating the operation of an electronic device(e.g., the electronic device 101) that generates a path rule throughuser log collection of the electronic device according to variousembodiments. Referring to FIG. 9 , the electronic device may generateand/or determine a path rule according to a user input, and may collectand store context information. The electronic device may assign apriority for providing a hint to each path rule based on the contextinformation so that an appropriate hint may be provided to a useroccasion.

In operation 910, the electronic device (e.g., the processor 120) mayreceive a user input.

According to an embodiment, the user input may be various, such as avoice input, a touch input, a user manipulation on buttons (or keys),and/or an input using a remote controller.

According to an embodiment, the user input may be a single input thatcauses the electronic device to perform one operation, or a plurality ofconsecutive inputs that causes the electronic device to performoperations consecutively.

In operation 920, the electronic device (e.g., the processor 120) maycollect context information. For example, the context information may bevarious, such as a current date, a current time, a location where theelectronic device is located, and/or schedule information stored in theelectronic device.

According to an embodiment, when the user input includes at least onevoice input via a microphone, the electronic device (e.g., the processor120) may collect utterance path rules of the electronic device based ona voice input in operation 930. For example, the electronic device maydetermine which utterance path rule is input as a voice in order for theelectronic device to perform specific operations sequentially, usingnatural language understanding. For example, the utterance path rule maybe log data for sequential operations of the electronic device.

In operation 940, the electronic device (e.g., the processor 120) maystore the collected utterance path rules in an utterance path rule tabletogether with the context information.

According to an embodiment, when the user input includes at least onetouch input on the touch screen, the electronic device (e.g., theprocessor 120) may generate a touch state log based on the touch inputin operation 950. For example, the electronic device may generate thetouch state log obtained by sequentially combining operations performedto correspond to each of the touch inputs which are consecutivelyreceived. For example, the electronic device may generate the touchstate log obtained by sequentially combining the states of theelectronic device which are changed by the touch input.

In operation 960, the electronic device (e.g., the processor 120) maydetermine a touch path rule based on the touch state log. For example,the electronic device may confirm to which portion a portioncorresponding to the state sequence log in which the touch state log isstored corresponds, by comparing the touch state log with a statesequence log stored in an existing rule database through a rule mapperprocess, and may extract the touch path rule corresponding to the touchstate log from the stored state sequence log. The rule mapper processhas been described in detail with reference to FIG. 8 , and thus furtherdescription thereof will be omitted.

In operation 970, the electronic device (e.g., the processor 120) maystore the touch path rule in the touch path rule table together with thecontext information.

For example, when a user has performed an operation of transmitting anew message to an acquaintance using SMS via the touch input in a car inthe morning, the electronic device may generate a log for a state of amessage application in the message application, that is, generate a pathrule, may analyze the path rule to extract the fact that the user hasperformed a path rule of “transmission of new message using SMS”, andmay store the corresponding path rule in the touch path rule table. Theelectronic device may also store context information indicating that theabove-described operation has been performed in the car in the morningwhile storing the path rule.

In operation 980, the electronic device (e.g., the processor 120) maygenerate a path rule by combining the utterance path rule of theutterance path rule table and the touch path rule of the touch path ruletable, and may store the generated path rule in an utterance/touchcombined path rule table together with the context information. Forexample, when the electronic device executes a first operation accordingto a first touch input, executes a second operation according to asecond touch input, executes a third operation according to a firstvoice input, and executes a fourth operation according to a third touchinput, the electronic device may store the first operation—the secondoperation—the third operation—the fourth operation as a single pathrule. For example, the electronic device may make a database togetherwith the number of times each path rule is performed or the like, andmay store the database. For example, the electronic device may alsostore information indicating whether each path rule has been performedthrough a touch or a voice input so as to be distinguished.

According to an embodiment, when a plurality of path rules is generatedthrough the above-described operations or when a plurality of path rulesis generated according to above-described operations 910 to 980repeatedly performed, the electronic device (e.g., the processor 120)may assign a priority for providing a hint to each path rule included inthe utterance/touch combined path rule table based on the contextinformation according to a designated condition, and may store theresult in a hint path rule table in operation 990.

According to an embodiment, the electronic device may give a hint to auser by assigning a priority to a path rule that the user usesfrequently in an existing touch method but does not use via voiceutterance, in order to expand user experience to the voice utterance.For example, a user frequently performs an operation of collectingphotos stored in a specific area in a gallery application to make theminto albums. Here, when the user mainly performs the operation ofcollecting photos through a touch and does not try to utter the samethrough a voice, the electronic device may provide the fact that thecorresponding function can be performed via the voice utterance to theuser as a hint, thereby helping to use frequently used functions througha voice command.

According to an embodiment, the electronic device may assign thepriority by classifying user input methods into touch input method andvoice input method according to the context information. For example,assuming that a user tries to execute a media content such as a movie,for example, when the user tries to execute the media content mainly viaa voice in a specific place such as a user's house other than a user'scompany and to execute the same via a touch in a place such as theuser's company, when the user is located in the company at the time ofdetermining a priority of a hint for inducing the voice utterance, apath rule for executing the media content may be set to have a lowerpriority in the hint, thereby reducing a probability of providing thehint.

Meanwhile, in the embodiment of FIG. 9 , it has been described that theelectronic device generates the path rule through user log collection ofthe electronic device. However, according to another embodiment, theelectronic device (e.g., the server 106) may receive operations executedby an external electronic device according to a user input or recordingof the operations from the external electronic device via a wirelesscommunication circuit, and may be operated in a method similar to theabove-described operation (corresponding method). For example, theelectronic device may execute operations of collecting utterance pathrules of the external electronic device, storing the utterance pathrules in the utterance path rule table together with the contextinformation of the external electronic device, generating the touchstate log of the external electronic device, determining the touch pathrule based on the touch state log, storing the touch path rule in thetouch path rule table together with the context information of theexternal electronic device, combining the utterance path rule and thetouch path rule to store the combined result in the utterance/touchcombined path rule table, and assigning a priority for providing a hintto each path rule included in the utterance/touch combined path ruletable based on the context information of the external electronicdevice, and may store the assigned result in the hint path rule table.For example, the electronic device may generate the path rule throughlog collection for each of a plurality of external devices (users),thereby having the hint path rule table for each of the plurality ofexternal devices (users).

FIG. 10 is a flowchart illustrating the operation of an electronicdevice (e.g., the electronic device 101) that generates a path rulethrough user log collection of the electronic device according tovarious embodiments. Referring to FIG. 10 , the electronic device maygenerate and/or determine a path rule according to a user input, and maycollect usage logs of any one application executed according to the userinput. The electronic device may assign a priority for providing a hintto each path rule based on the usage logs of the application so that thehint of the application may be appropriately provided.

In operation 1005, the electronic device (e.g., the processor 120) mayreceive a user input.

According to an embodiment, the user input may be various, such as avoice input, a touch input, a user manipulation on buttons (or keys),and/or an input using a remote controller.

According to an embodiment, the user input may be a single input thatcauses the electronic device to perform one operation, or a plurality ofconsecutive inputs that causes the electronic device to performoperations consecutively.

In operation 1010, the electronic device (e.g., the processor 120) maycollect the usage logs of the application executed according to the userinput.

According to an embodiment, the usage logs of the application mayinclude context information. For example, the usage logs of theapplication may include information such as at which time theapplication is used, at which place the application is used, in whichcontext the application is used, and/or with which application theapplication is used together.

According to an embodiment, the electronic device may aggregate thecollected usage logs of each of the applications for a specific period.For example, the electronic device may aggregate the usage logs of eachof the applications in various units, such as daily, monthly, hourly, orseasonal units. By way of another example, the electronic device maymanage user characteristic data as a result of the aggregation of theusage logs of each of the applications for the specific period bydefining the specific period. For example, when a user uses a firstapplication during the weekday morning for the specific period but doesnot use the first application during the weekend, the electronic devicemay manage the user characteristic data as the fact that the user mainlyuses the first operation during the weekday morning, based on the resultof the aggregation of the usage logs of the first application for thespecific period. By way of still another example, the electronic devicemay manage the use logs of the first application by dividing a pluralityof periods into a period A, a period B, and the like. For example, theelectronic device may manage the usage logs of the first application bydividing the database into the aggregation for a week and theaggregation for a day, and may utilize the managed usage logs for hintgeneration.

According to an embodiment, when the user input includes at least onevoice input via a microphone, the electronic device (e.g., the processor120) may collect utterance path rules of the electronic device based onthe voice input in operation 1015. For example, the electronic devicemay determine which utterance path rule is input as a voice in order forthe electronic device to perform specific operations sequentially, usinga natural language understanding technique. For example, the utterancepath rule may be log data for sequential operations of the electronicdevice.

In operation 1020, the electronic device (e.g., the processor 120) maystore the collected utterance path rules in an utterance path ruletable.

According to an embodiment, when the user input includes at least onetouch input on the touch screen, the electronic device (e.g., theprocessor 120) may generate a touch state log based on the touch inputin operation 1025. For example, the electronic device may generate thetouch state log obtained by sequentially combining operations performedto correspond to each of the touch inputs which are consecutivelyreceived. For example, the electronic device may generate the touchstate log obtained by sequentially combining the states of theelectronic device which are changed by the touch input.

In operation 1030, the electronic device (e.g., the processor 120) maydetermine a touch path rule of an application based on the touch statelog. For example, the electronic device may confirm to which portion aportion corresponding to the state sequence log in which the touch statelog is stored corresponds, by comparing the touch state log with a statesequence log stored in an existing rule database through a rule mapperprocess, and may extract the touch path rule corresponding to the touchstate log from the stored state sequence log. The rule mapper processhas been described in detail with reference to FIG. 8 , and thus furtherdescription thereof will be omitted.

In operation 1035, the electronic device (e.g., the processor 120) maystore the touch path rule in the touch path rule table of theapplication.

In operation 1040, the electronic device (e.g., the processor 120) maycombine the utterance path rule of the utterance path rule table and thetouch path rule of the touch path rule table, and may store the combinedresult in an utterance/touch combined path rule table. For example, theelectronic device executes a first operation according to a first touchinput, executes a second operation according to a second touch input,executes a third operation according to a first voice input, andexecutes a fourth operation according to a third touch input, theelectronic device may store the first operation—the second operation—thethird operation—the fourth operation as a single path rule. For example,the electronic device may make a database together with the number oftimes each path rule is performed or the like, and may store thedatabase. For example, the electronic device may also store informationindicating whether each path rule has been performed through a touch ora voice input so as to be distinguished.

In operation 1045, the electronic device (e.g., the processor 120) mayupdate a daily aggregate table of the utterance/touch path rule based onthe combined path rule. For example, the electronic device may aggregatedaily generated path rules and may store the aggregated path rules inthe daily aggregate table.

In operation 1050, the electronic device (e.g., the processor 120) maygenerate a table of the utterance/touch path rule of the period A basedon the daily aggregation table, or may update the previously generatedtable of the utterance/touch path rule of the period A. For example, theelectronic device may generate the table of the utterance/touch pathrule of the period A by collecting the path rules stored in the dailyaggregation tables corresponding to the period A, in order to manage thepath rules during the period A. By way of another example, theelectronic device may update the previously generated table ofutterance/touch path rule of the period A using the path rules stored inthe daily aggregation tables corresponding to the period A.

In operation 1055, the electronic device (e.g., the processor 120) maygenerate a user's preference path rule to which a priority score isassigned based on the usage logs of the application and a table of thepreference path rule of the period A including time, place, occasion,application (TPOA) information corresponding to the user's preferencepath rule, or may update a previously generated preference path ruletable of the period A. For example, the electronic device may assign thepriority score to each path rule included in the table of theutterance/touch path rule of the period A based on the usage logs(referred to as user's profile information) of the first application.For example, the TPOA information may be usage-based information, andmay be a generation time of the path rule, a generation location of thepath rule, a context when the path rule is generated, and/or applicationinformation of the path rule.

In operation 1060, the electronic device (e.g., the processor 120) maygenerate a table of an utterance/touch path rule of the period B basedon the daily aggregation table, or may update a previously generatedtable of an utterance/touch path rule of the period B. For example, theelectronic device may generate the table of the utterance/touch pathrule of the period B by collecting the path rules stored in the dailyaggregation tables corresponding to the period B, in order to manage thepath rules during the period B.

In operation 1065, the electronic device (e.g., the processor 120) maygenerate a user's preference path rule to which a priority score isassigned based on the usage logs of the application and a table of thepreference path rule of the period B including TPOA informationcorresponding to the user's preference path rule, or may update apreviously generated preference path rule table of the period B. Forexample, the electronic device may assign the priority score to eachpath rule included in the table of the utterance/touch path rule of theperiod B based on the usage logs (referred to as user's profileinformation) of the first application. For example, the TPOA informationmay be usage-based information, and may be a generation time of the pathrule, a generation location of the path rule, a context when the pathrule is generated, and/or application information of the path rule.

According to an embodiment, when the period A is set as six months, theperiod B is set as one week, and a larger priority score is assigned tothe period A rather than the period B, path rules preferred by a user,that is, path rules corresponding to a hint to be provided to the usermay be path rules that are used frequently for the six months.

According to another embodiment, when the period A is set as six months,the period B is set as one week, and a larger priority score is assignedto the period B rather than the period A, path rules preferred by a usermay be path rules that are frequently used recently, and the path rulesincluded in the period A may be included in an upper recommended pathrule with a higher priority score in a user preference rule list.

In operation 1070, the electronic device (e.g., the processor 120) maygenerate a final preference path rule table based on the preference pathrule table of the period A and the preference path rule table of theperiod B, or may update a previously generated final preference pathrule table. For example, the electronic device may generate the finalpreference path rule table including the preference path rule table ofthe period A, the path rules included in the period B, and the TPOAinformation. For example, the electronic device may manage the pathrules included in the final preference path rule table together with theTPOA information to provide a hint according to a user occasion. Forexample, the electronic device may recommend a hint about a functionthat is frequently used in the user occasion using the TPOA information,and when the user has difficulty in inputting the voice input, thefunction may be executed according to the touch input of the user withrespect to the provided hint.

Meanwhile, according to the above-described embodiment, it has beendescribed that the electronic device aggregates the generated path rulesin daily units in operation 1045, but according to another embodiment,the electronic device may aggregate and store the path rules generatedduring various unit periods such as monthly, hourly, seasonal units, andthe like.

In the above-described embodiment of FIG. 10 , in regard to thecollection of the usage logs of the application, it has been describedthat the electronic device collects the usage logs of the applicationexecuted according to the user input in operation 1010, but according toanother embodiment, the electronic device may collect the usage logs ofthe application during the execution of the operation of the electronicdevice and/or the execution of the application.

In the above-described embodiment of FIG. 10 , it has been describedthat the electronic device generates the path rule through the user logcollection of the electronic device. However, according to anotherembodiment, the electronic device (e.g., the server 106) may receive,from an external electronic device via a wireless communication circuit,operations executed by the external electronic device according to theuser input or recording of the operations and the usage log of the firstapplication, and may be operated in a method similar to theabove-described operation (corresponding method). For example, theelectronic device may execute operations of collecting utterance pathrules of the external electronic device, storing the utterance pathrules in the utterance path rule table, generating a touch state log ofthe external electronic device, determining a touch path rule based onthe touch state log, storing the touch path rule in the touch path ruletable, combining the utterance path rule and the touch path rule tostore the combined result in the utterance/touch combined path ruletable, updating the daily aggregation table of the utterance/touch pathrule based on the combined touch path rule, generating the table of theutterance/touch path rule of the period A, generating the preferencepath rule table of the period A, generating the table of theutterance/touch path rule of the period B, generating the preferencepath rule table of the period B, and generating the final preferencepath rule table based on the preference path rule table of the period Aand the preference path rule table of the period B. For example, theelectronic device may generate the path rule through log collection foreach of a plurality of external devices (users), thereby having thefinal preference path rule table for each of the plurality of externaldevices (users).

FIG. 11 is a flowchart illustrating a hint generation operation forexecution of an electronic device (e.g., the electronic device 101)according to various embodiments. Referring to FIG. 11 , the electronicdevice may generate a domain-specific hint that is effective for the useof the electronic device in the processor (e.g., the processor 120) ofthe electronic device based on a usage pattern of a user, and mayprovide the generated hint to the user.

In operation 1110, the electronic device (e.g., the processor 120) maycollect logs of the electronic device.

According to an embodiment, the electronic device may collect the logsof the electronic device. For example, when executing at least oneoperation according to a user input, the electronic device may collectlogs according to the execution of the at least one operation. Forexample, the electronic device may collect screen transformationinformation displayed when a user uses the electronic device.

According to an embodiment, the logs may include recording of screenswhich are sequentially displayed by the electronic device in order forthe electronic device to execute a function including the at least oneoperation according to the user input and/or recording of operationswhich are sequentially executed by the electronic device. For example, asingle screen displayed according to one operation executed by theelectronic device or the execution of the one operation may be referredto as a state, and recording of the operations sequentially executed bythe electronic device or recording of screens sequentially displayed tocorrespond to the sequentially executed operations may be referred to asa state sequence log (hereinafter, also referred to as a sequence ofstates).

According to an embodiment, the logs may further include information onthe user input. For example, the user input may be various, such as avoice input using a microphone, a touch input on a touch screen, a usermanipulation on buttons (or keys), and/or an input using a remotecontroller.

According to an embodiment, the logs may further include contextinformation (also referred to as context) when the electronic devicereceives, from a user, a user input that causes execution of any onefunction of the electronic device. For example, the context informationmay be various, such as a location, a date, a time, and/or scheduleinformation stored in the electronic device.

In operation 1120, the electronic device (e.g., the processor 120) mayanalyze collected logs and may detect at least one path rule indicatingsequentially changed states.

According to an embodiment, the electronic device may select statescorresponding to the use intention or purpose of the electronic devicein such a manner that the collected logs are wholly or partially matchedwith a previously stored state sequence log (also referred to as areference sequence of states), and may find the path rule allowing theselected states to be sequentially executed. For example, the previouslystored state sequence log may be updated according to the operation ofthe electronic device depending on the user input. For example,according to the detection of the path rule, the electronic device mayconfirm which function of the electronic device is used, in whichcontext the function is used, and/or through which input means thefunction is used.

According to another embodiment, the electronic device may analyze astate log that is repeatedly used through pattern analysis of thecollected state logs to thereby extract a new path rule. For example,when the state log is performed a designated reference number of timesor more, the electronic device may confirm that a user has repeatedlyperformed the above-described function through analysis of the collectedstate logs, and may newly acquire a path rule including the statesequence.

In operation 1130, the electronic device (e.g., the processor 120) mayclassify the detected path rules for each domain. For example, theelectronic device may classify to which domain the detected path rulebelongs.

According to an embodiment, the electronic device may classify thedomains in functional units. For example, the domains may be classifiedinto messenger domains, game domains, and Internet banking domains, andthe like. For example, the messenger domain may include path rulesperformed in KakaoTalk, Line Messenger, Allo, Facebook Messenger, andthe like.

According to another embodiment, the electronic device may classify thedomains in application units. For example, the electronic device maydetermine whether the corresponding path rule belongs to KakaoTalk,Line, or the like.

In operation 1140, the electronic device (e.g., the processor 120) maydetermine a priority of the path rules which are classified for eachdomain and may select the path rule based on the priority for eachdomain.

According to an embodiment, the electronic device may change thepriority of the path rules classified for each domain corresponding to ahint to be provided to a user based on information included in the logs.

According to an embodiment, the priority may be determined so that ahint converted into the natural language corresponding to the path ruleclassified for each domain can be provided to a user according to a useroccasion in various ways. For example, the electronic device maydetermine the user occasion based on the context information.

According to an embodiment, when the path rules classified for each of aplurality of domains are detected through the log analysis, theelectronic device may determine the priority of the plurality of pathrules in various ways depending on the purpose in which the plurality ofpath rules provides the hint to the user. For example, when there is thepurpose to provide the hint so that the user may easily process the pathrule, which is frequently used via a touch input, in the naturallanguage, a high priority may be assigned to the path rule which is usedvia the touch input but is not utilized well via the voice input. Thiscan increase the utilization of the hint by reducing interface steps ofa frequently used function. By way of another example, assuming that auser uses the same path rule mainly through the voice input at home andcontrols the same path rule via the touch input at the office, when auser's location is the office at the time of providing a hint to theuser, the electronic device may exclude the path rule of the hint thatsuggests the voice input to the user.

In operation 1150, the electronic device (e.g., the processor 120) mayconvert the path rule selected based on the priority for each domaininto a natural language hint.

According to an embodiment, the electronic device may convert the pathrule selected based on the priority for each domain into a hint in theform of the natural language through a natural language generationtechnique or mapping with standard utterance matched with the path rule.

In operation 1160, the electronic device (e.g., the processor 120) mayprovide the hint converted into the natural language.

According to an embodiment, the electronic device may provide the hintconverted into the natural language via a display.

According to another embodiment, the electronic device may output thehint converted into the natural language as a voice via a speaker, usinga character-to-speech conversion technique.

According to still another embodiment, the electronic device may providethe hint converted into the natural language via the display and thespeaker.

According to an embodiment, when a user utters or touches the hintdisplayed on the display, the electronic device may sequentially executethe states of the path rule corresponding to the hint. For example, theelectronic device may display a final state in which the states aresequentially executed on the touch screen. By way of another example,the electronic device may sequentially display all the sequentialexecutions of the states on the touch screen.

According to an embodiment, the electronic device may automaticallyexecute the above-described operations of FIG. 11 upon the execution ofthe electronic device, or may execute the above-described operations ofFIG. 11 according to the user input such as a touch input, a voiceinput, and/or a user manipulation on buttons (or keys) which may causethe above-described operations of FIG. 11 to be executed.

Meanwhile, in the above-described embodiment of FIG. 11 , it has beendescribed that the electronic device executes all of operations ofproviding the hint converted into the natural language through logcollection of the electronic device. However, according to anotherembodiment, the electronic device (e.g., the server 106) may receiveoperations executed by an external electronic device according to theuser input or recording of the operations from the external electronicdevice via a wireless communication circuit, and may be operated in amethod similar to the above-described operation (corresponding method).For example, the electronic device may execute operations of collectinglogs of the external electronic device, analyzing the collected logs ofthe external electronic device to detect the path rule, classifying thedetected path rules for each domain, determining a priority of the pathrules classified for each domain, selecting the path rule based on thepriority for each domain, and converting the selected path rule into thenatural language hint. In addition, the electronic device may transmitthe hint converted into the natural language to the external electronicdevice via a wireless communication circuit.

FIG. 12 is a flowchart illustrating the operation of an electronicdevice (e.g., the electronic device 101) that generates a path rulethrough user log collection of the electronic device according tovarious embodiments.

In operation 1210, the electronic device (e.g., the processor 120) maycollect application state logs in which an input means is classified.

According to an embodiment, when the electronic device executes at leastone operation of an application via a user's voice input using amicrophone, via a user's touch input using a touch screen, or via amixed input of user's voice and touch, the electronic device mayclassify the corresponding input means.

According to an embodiment, a single screen displayed according to oneoperation executed by the electronic device or the execution of the oneoperation may be referred to as a state, and recording of operationsexecuted by the electronic device or recording of screens displayed tocorrespond to the executed operations may be referred to as a state log.

In operation 1220, the electronic device (e.g., the processor 120) maydetermine a path rule according to the application state log through therule mapper process.

According to an embodiment, the electronic device may confirm to whichportion a portion where the application state log corresponds to thestate sequence log corresponds, by comparing the application state logwith a state sequence log stored in an existing rule database throughthe rule mapper process, and may determine the touch path rulecorresponding to the application state log from the state sequence log.

According to an embodiment, the path rule may include information of theinput means so that it can be known through which interface a userperforms an input.

According to an embodiment, when the path rule is an utterance pathrule, the electronic device (e.g., the processor 120) may store the pathrule in an utterance path rule table in operation 1230. For example,when the information of the input means included in the path rule is avoice input using a microphone, the electronic device may determine thatthe path rule is the utterance path rule.

According to an embodiment, when the path rule is a touch path rule, theelectronic device (e.g., the processor 120) may store the path rule in atouch path rule table in operation 1240. For example, when theinformation of the input means included in the path rule is a touchinput using a touch screen, the electronic device may determine that thepath rule is the touch path rule.

According to an embodiment, when the path rule is a touch and utterancepath rule, the electronic device (e.g., the processor 120) may store thepath rule in a touch and utterance path rule table in operation 1250.For example, when the information of the input means included in thepath rule is the voice input using the microphone and the touch inputusing the touch screen, the electronic device may determine that thepath rule is the touch and utterance path rule.

In operation 1260, the electronic device (e.g., the processor 120) mayconfirm a domain including the path rule.

According to an embodiment, the electronic device may classify to whichdomain the path rule belongs. For example, the domains may be classifiedinto messenger domains, game domains, and Internet banking domains, andthe like. For example, the messenger domain may include path rulesperformed in KakaoTalk, Line Messenger, Allo, Facebook Messenger, andthe like.

According to another embodiment, the electronic device may classify thedomains in application units. For example, the electronic device maydetermine whether the corresponding path rule belongs to KakaoTalk orLine.

In operation 1270, the electronic device (e.g., the processor 120) maycombine the path rule of the utterance path rule table, the path rule ofthe touch path rule table, and the touch and utterance path rule of thetouch and utterance path rule table, and may store the combined resultin an utterance/touch combined path rule table. In addition, theelectronic device may also store to which at least one domain thecombined path rule belongs in the utterance/touch path rule table.

According to an embodiment, when a plurality of path rules is generatedthrough the above-described operations or when a plurality of path rulesis generated according to above-described operations 1210 to 1270repeatedly performed, the electronic device (e.g., the processor 120)may assign a priority for providing a hint to each path rule for eachdomain which is included in the utterance/touch combined path rule tableaccording to a designated condition, and may store the result in a hintpath rule table in operation 1280. For example, the designated conditionmay be a frequency of use of each path rule for each domain.

According to an embodiment, it may be assumed that there are anapplication used by a user in the electronic device and a system thatprovides a function frequently used in the corresponding application asa hint. The electronic device may determine a domain of an applicationof a context in which a call is made, and may provide the determineddomain as a hint corresponding to the path rule which is actually usedmost frequently by a user in the domain of the corresponding applicationincluded in the path rule table (e.g., the utterance/touch combined pathrule table). For example, assuming that the path rule which is mostfrequently used by the user in a gallery application domain is “deletionof the most recent photo” or “album creation”, when the electronicdevice executes a gallery application according to a user input andcalls an artificial intelligent agent for allowing the electronic deviceto provide a hint, the electronic device may provide “Do you want todelete the most recent photo?” or “Do you want to create a recentalbum?” each corresponding to the path rule as the hint. By way ofanother example, the path rules which have been used most frequently inthe corresponding domain may be provided to the user as hints in anotherdomain.

Meanwhile, in the above-described embodiment of FIG. 12 , it has beendescribed that the electronic device generates the path rule throughuser log collection of the electronic device. However, according toanother embodiment, the electronic device (e.g., the server 106) mayreceive operations executed by the external electronic device accordingto the user input or recording of the operations from the externalelectrode device via the wireless communication circuit, and may beoperated in a method similar to the above-described operation(corresponding method). For example, the electronic device may executeoperations of collecting the application state log in which input meansof the external electronic device is classified, determining the pathrule, storing the path rule in the utterance path rule table, the touchpath rule table, and/or the touch and utterance path rule table,confirming the domain including the path rule, combining the path rulesto store the combined result in the utterance/touch combined path ruletable, and assigning the priority for providing the hint to each pathrule for each domain included in the utterance/touch combined path ruletable to store the result in the hint path rule table.

FIG. 13 is a flowchart illustrating the operation of an electronicdevice (e.g., the electronic device 101) that generates a path rulethrough user log collection of the electronic device according tovarious embodiments. Referring to FIG. 13 , the electronic device maystore frequencies and types of parameters generated by at least oneoperation and may use the stored frequencies and types thereof ingenerating a hint corresponding to a path rule. For example, theparameters may be information on the operation of the electronic deviceand information for executing the operation, and may be an input valueof a user reflecting the intention of the user. For example, theparameter in the message application may be a message content or arecipient. As another example, the parameter may be a search target in aspecific application.

In operation 1310, the electronic device (e.g., the processor 120) maycollect application state logs in which an input means is classified.

According to an embodiment, when the electronic device executes at leastone operation of an application via a user's voice input using amicrophone, via a user's touch input using a touch screen, or via amixed input of user's voice and touch, the electronic device mayclassify the corresponding input means.

According to an embodiment, a single screen displayed according to oneoperation executed by the electronic device or the execution of the oneoperation may be referred to as a state, and recording of operationsexecuted by the electronic device or recording of screens displayed tocorrespond to the executed operations may be referred to as a state log.

In operation 1320, the electronic device (e.g., the processor 120) maycollect parameters generated in each state. For example, the electronicdevice may collect the parameters according to the user input which aregenerated according to a change in the states together with collectionof the state log, and may store the collected parameters.

In operation 1330, the electronic device (e.g., the processor 120) maydetermine a path rule according to the application state log through arule mapper process.

According to an embodiment, the electronic device may confirm to whichportion a portion where the application state log corresponds to thestate sequence log corresponds, by comparing the application state logwith a state sequence log stored in an existing rule database throughthe rule mapper process, and may determine the path rule correspondingto the application state log from the state sequence log.

According to an embodiment, the path rule may include information of theinput means so that it can be known through which interface a userperforms an input.

According to an embodiment, the path rule of the electronic device mayinclude the collected parameters.

According to an embodiment, when the path rule is an utterance pathrule, the electronic device (e.g., the processor 120) may store the pathrule in an utterance path rule table in operation 1340. In addition,when storing the path rule, the electronic device may also store thecollected parameters. For example, when information of the input meansincluded in the path rule is a voice input using a microphone, theelectronic device may determine that the path rule is the utterance pathrule.

According to an embodiment, when the path rule is a touch path rule, theelectronic device (e.g., the processor 120) may store the path rule in atouch path rule table in operation 1350. In addition, when storing thepath rule, the electronic device may also store the collectedparameters. For example, when the information of the input meansincluded in the path rule is a touch input using a touch screen, theelectronic device may determine that the path rule is the touch pathrule.

According to an embodiment, when the path rule is a touch and utterancepath rule, the electronic device (e.g., the processor 120) may store thepath rule in a touch and utterance path rule table in operation 1360. Inaddition, when storing the path rule, the electronic device may alsostore the collected parameters. For example, when the information of theinput means included in the path rule is the voice input using themicrophone and the touch input using the touch screen, the electronicdevice may determine that the path rule is the touch and utterance pathrule.

In operation 1370, the electronic device (e.g., the processor 120) maycombine the path rule of the utterance path rule table, the path rule ofthe touch path rule table, and the touch and utterance path rule of thetouch and utterance path rule table, and may store the combined resultin an utterance/touch combined path rule table. In addition, whenstoring the path rule, the electronic device may also store thecollected parameters.

According to an embodiment, when a plurality of path rules is generatedthrough the above-described operations or when a plurality of path rulesis generated according to above-described operations 1310 to 1370repeatedly performed, the electronic device may assign a priority forproviding a hint to each path rule which is included in theutterance/touch combined path rule table according to a designatedcondition, and may store the result in a hint path rule table inoperation 1380.

For example, when a user speaks “Send message saying that I am latetoday to J” as a voice utterance, the electronic device may ask furtherquestions in order to acquire a parameter from a user while performing apath rule for sending a message, or may receive an application that is ameans for sending a message through an input field from a user. Forexample, when a user uses “Kakao Talk” as the means for sending amessage and “Kakao Talk” is repeatedly used as the means for sending amessage, the electronic device may allow the parameter to be included inthe path rule, and may generate a hint such as “Would you like to send amessage saying that I am late today to J using Kakao Talk?”.

The following Table 1 is an example of the parameters that can be usedrepresentatively in applications according to various embodiments.

TABLE 1 Parameter Applications targets Parameter names Utilized hintexamples Galley search gallery_search_keyword Find travel photoapplication Message search messages_search_keyword Find characterincluding application coupon Message Messages_contack_name[name] Sendphoto taken transmission Messages_contact_id(messages_ yesterday toHong-gil-dong target phone_number[phone number]) Calendar SearchCalendar_search_keyword Find meeting schedule application Time Timertime Clock_timer Start timer for two minutes application Camera Usagemode Camera_change_mode Take picture in food mode application E-mailSearch Email_search_usemame[Sender] Show me message from A application

In the above-described embodiment of FIG. 13 , in regard to thecollection of the parameters, it has been described that the electronicdevice collects the parameters generated in each state in operation1320. However, according to another embodiment, the electronic devicemay collect the parameters during the operation of the electronicdevice.

In the above-described embodiment of FIG. 13 , it has been describedthat the electronic device generates the path rule through thecollection of the user logs of the electronic device. However, accordingto another embodiment, the electronic device (e.g., the server 106) mayreceive, from an external electronic device via a wireless communicationcircuit, operations executed by the external electronic device accordingto the user input or recording of the operations and parameterinformation, and may be operated in a method similar to theabove-described operation (corresponding method). For example, theelectronic device may execute operations of collecting the state logs ofthe application in which an input means of the external electronicdevice is classified, collecting the parameters generated in each state,determining the path rule of the external electronic device, storing thepath rules in the utterance path rule table, the touch path rule table,and/or the touch and utterance path rule table, combining the path rulesto store the combined result in the utterance/touch combined path ruletable, assigning the priority for providing the hint to each path ruleincluded in the utterance/touch combined path rule table, and generatingthe hint corresponding to the path rule based on the parameter to storethe generated hint in the hint path rule table. For example, theelectronic device may generate the path rules through the collection ofthe state logs of the application for each of the plurality of externaldevices (users), thereby having the hint path rule table for each of theplurality of external devices (users).

FIG. 14 is a flowchart illustrating the operation of an electronicdevice (e.g., the electronic device 101) that generates a path rulethrough user log collection of the electronic device according tovarious embodiments. Referring to FIG. 14 , the electronic device maygenerate and/or determine a path rule according to a user input, and maycollect and store context information. The electronic device may assigna priority for providing a hint to each path rule based on the contextinformation and may provide a hint suitable for a user occasion.

In operation 1410, the electronic device (e.g., the processor 120) maycollect application state logs in which an input means is classified,together with the context information.

According to an embodiment, the context information may be various, suchas a current date, a current time, a location where the electronicdevice is located, and/or schedule information stored in the electronicdevice.

According to an embodiment, when the electronic device executes at leastone operation of an application via a user's voice input using amicrophone, via a user's touch input using a touch screen, or via amixed input of user's voice and touch, the electronic device mayclassify the corresponding input means.

According to an embodiment, a single screen displayed according to oneoperation executed by the electronic device or the execution of the oneoperation may be referred to as a state, and recording of operationsexecuted by the electronic device or recording of screens displayed tocorrespond to the executed operations may be referred to as a state log.

In operation 1420, the electronic device (e.g., the processor 120) maydetermine a path rule according to the application state log through arule mapper process.

According to an embodiment, the electronic device may confirm to whichportion a portion where the application state log corresponds to thestate sequence log corresponds, by comparing the application state logwith a state sequence log stored in an existing rule database throughthe rule mapper process, and may determine the path rule correspondingto the application state log from the state sequence log.

According to an embodiment, the path rule may include information of theinput means so that it can be known through which interface a userperforms an input.

According to an embodiment, when the path rule is an utterance pathrule, the electronic device (e.g., the processor 120) may store the pathrule in an utterance path rule table together with the contextinformation in operation 1430. For example, when information of theinput means included in the path rule is a voice input using amicrophone, the electronic device may determine that the path rule isthe utterance path rule.

According to an embodiment, when the path rule is a touch path rule, theelectronic device (e.g., the processor 120) may store the path rule in atouch path rule table together with the context information in operation1440. For example, when the information of the input means included inthe path rule is a touch input using a touch screen, the electronicdevice may determine that the path rule is the touch path rule.

According to an embodiment, when the path rule is a touch and utterancepath rule, the electronic device (e.g., the processor 120) may store thepath rule in a touch and utterance path rule table together with thecontext information in operation 1450. For example, when the informationof the input means included in the path rule is the voice input usingthe microphone and the touch input using the touch screen, theelectronic device may determine that the path rule is the touch andutterance path rule.

According to an embodiment, assuming that a user inputs a new message tobe transmitted to an acquaintance via SMS in the company, when the userfirst starts with voice utterance and composes the contents of themessage through the touch input, the electronic device may collect statelogs in a message application, may determine that a path rule of“transmission of new message via SMS” has been performed according tothe above-described operations, and may store the determined path rulein the touch and utterance path rule table. In addition, the electronicdevice may also store context information of a place such as the companyin the touch and utterance path rule table.

In operation 1460, the electronic device (e.g., the processor 120) maycombine the path rule of the utterance path rule table, the path rule ofthe touch path rule table, and the touch and utterance path rule of thetouch and utterance path rule table, and may store the combined resultin an utterance/touch combined path rule table together with the contextinformation. For example, the electronic device may also make a databasetogether with the number of times (frequency) the path rule is performedby a user, and may store the database. For example, the electronicdevice may also store information indicating whether the path rule hasbeen performed through the touch input or the voice input, or whetherthe path rule have been performed through a combination of the twoinputs. For example, the electronic device may also store informationindicating which user input (referred to as input means or interface)has been applied in each state of the path rule so as to bedistinguished.

According to an embodiment, when a plurality of path rules is generatedthrough the above-described operations or when a plurality of path rulesis generated according to above-described operations 1410 to 1460repeatedly performed, the electronic device (e.g., the processor 120)may assign a priority for providing a hint to each path rule included inthe utterance/touch combined path rule table based on the contextinformation according to a designated condition, and may store theresult in a hint path rule table in operation 1470.

Meanwhile, in the above-described embodiment of FIG. 14 , it has beendescribed that the electronic device generates the path rule throughuser log collection of the electronic device. However, according toanother embodiment, the electronic device (e.g., the server 106) mayreceive, from the external electronic device via a wirelesscommunication circuit, operations executed by an external electronicdevice or recording of the operations according to the user input andthe context information, and may be operated in a method similar to theabove-described operation (corresponding method). For example, theelectronic device may execute operations of collecting the state logs ofthe application in which an input means of the external electronicdevice is classified, together with the context information of theexternal electronic device, determining the path rule of the externalelectronic device, storing the path rules in the utterance path ruletable, the touch path rule table and/or the touch and utterance pathrule table, combining the path rules to store the combined result in theutterance/touch combined path rule table, and assigning the priority forproviding the hint to each path rule included in the utterance/touchcombined path rule table based on the context information to store theresult in the hint path rule table. For example, the electronic devicemay generate the path rules through the collection of the state logs ofthe application for each of the plurality of external devices (users),thereby having the hint path rule table for each of the plurality ofexternal devices (users).

FIG. 15 is a flowchart illustrating the operation of an electronicdevice (e.g., the electronic device 101) that generates a path rulethrough user log collection of the electronic device according tovarious embodiments. Referring to FIG. 15 , the electronic device maygenerate and/or determine a path rule according to a user input, and maygenerate a macro designation suggestion hint for a hint corresponding tothe path rule.

In operation 1510, the electronic device (e.g., the processor 120) maycollect application state logs in which an input means is classified.

According to an embodiment, when the electronic device executes at leastone operation of an application via a user's voice input using amicrophone, via a user's touch input using a touch screen, or via amixed input of user's voice and touch, the electronic device mayclassify the corresponding input means.

According to an embodiment, a single screen displayed according to oneoperation executed by the electronic device or the execution of the oneoperation may be referred to as a state, and recording of operationsexecuted by the electronic device or recording of screens displayed tocorrespond to the executed operations may be referred to as a state log.

According to an embodiment, the electronic device may collect contextinformation together with the state logs. For example, the contextinformation may be various, such as a current date, a current time, alocation where the electronic device is located, and/or scheduleinformation stored in the electronic device.

In operation 1520, the electronic device (e.g., the processor 120) maydetermine a path rule according to the application state log through therule mapper process.

According to an embodiment, the electronic device may confirm to whichportion a portion where the application state log corresponds to thestate sequence log corresponds, by comparing the application state logwith a state sequence log stored in an existing rule database throughthe rule mapper process, and may determine the path rule correspondingto the application state log from the state sequence log.

According to an embodiment, the path rule may include information of theinput means so that it can be known through which interface a userperforms an input.

According to an embodiment, when the path rule is an utterance pathrule, the electronic device (e.g., the processor 120) may store the pathrule in an utterance path rule table in operation 1530. For example,when the information of the input means included in the path rule is avoice input using a microphone, the electronic device may determine thatthe path rule is the utterance path rule. For example, the electronicdevice may store the context information in the utterance path ruletable together with the path rule.

According to an embodiment, when the path rule is a touch path rule, theelectronic device (e.g., the processor 120) may store the path rule in atouch path rule table in operation 1540. For example, when theinformation of the input means included in the path rule is a touchinput using a touch screen, the electronic device may determine that thepath rule is the touch path rule. For example, the electronic device maystore the context information in the touch path rule table together withthe path rule.

According to an embodiment, when the path rule is a touch and utterancepath rule, the electronic device (e.g., the processor 120) may store thepath rule in a touch and utterance path rule table in operation 1550.For example, when the information of the input means included in thepath rule is the voice input using the microphone and the touch inputusing the touch screen, the electronic device may determine that thepath rule is the touch and utterance path rule. For example, theelectronic device may store the context information in the touch andutterance path rule table together with the path rule.

In operation 1560, the electronic device (e.g., the processor 120) maycombine the path rule of the utterance path rule table, the path rule ofthe touch path rule table, and the touch and utterance path rule of thetouch and utterance path rule table, and may store the combined resultin an utterance/touch combined path rule table. For example, theelectronic device may further store the context information together.For example, the electronic device may make a database together with thenumber of times (frequency) the path rule is performed by a user, andmay store the database. For example, the electronic device may alsostore information indicating whether the path rule has been performedvia the touch input or the voice input, or whether the path rule hasbeen performed via a combination of the two inputs. For example, theelectronic device may also store information indicating which the userinput (referred to as input means or interface) has been applied in eachstate of the path rule so as to be distinguished.

In operation 1570, the electronic device (e.g., the processor 120) maygenerate a macro designation suggestion hint for a hint corresponding tothe path rule.

According to an embodiment, the electronic device may provide a macrofunction that can replace the hint corresponding to the path rule with apersonalized short sentence. For example, assuming that there is a hintcapable of executing a path rule of “Make photos taken yesterday with Jinto an album and store the album in a cloud”, when the hint isdesignated as a macro such as “J album” through setting and a userutters the macro, the electronic device may perform the correspondingfunction even if the user does not utter the hint corresponding to theentire path rule.

According to an embodiment, the electronic device or the processor(e.g., the processor 120) of the electronic device may include a macrosuggestion hint generator (not shown). The macro suggestion hintgenerator may generate a hint that helps the user to easily set themacro corresponding to the hint of the path rule. For example, the macrosuggestion hint generator may recommend, to the user, a path capable ofdesignating a macro with a designated criterion. For example, thedesignated criterion may be various, such as the number of times thepath rule is performed and/or complexity of the path rule.

For example, when the path rule indicating that a user makes the photostaken yesterday with J into an album and stores the album in a cloud isused a designated reference number of times or more, the electronicdevice may convert the corresponding path rule into a sentence formthrough a natural language generation technique or the like, and mayprovide the result as a hint. In addition, the electronic device maygenerate a macro designation suggestion hint such as “a function ofmaking the photos taken yesterday with J into an album and storing thealbum in a cloud can be macro-stored and quickly performed” whichsuggests designating a macro for the hint, and may allow the user toeasily designate the macro for the hint.

Meanwhile, in the above-described embodiment of FIG. 15 , it has beendescribed that the electronic device generates the path rule throughuser log collection of the electronic device. However, according toanother embodiment, the electronic device (e.g., the server 106) mayreceive, from an external electronic device via a wireless communicationcircuit, operations executed by the external electronic device accordingto the user input or recording of the operations, and may be operated ina method similar to the above-described operation (correspondingmethod). For example, the electronic device may execute operations ofcollecting the state logs of the application in which an input means ofthe external electronic device is classified, determining the path ruleof the external electronic device, storing the path rules in theutterance path rule table, the touch path rule table, and/or the touchand utterance path rule table, combining the path rules to store thecombined result in the utterance/touch combined path rule table, andgenerating the hint for suggesting the macro designation for the hintcorresponding to the path rule.

FIG. 16 is a flowchart illustrating the operation of an electronicdevice (e.g., the electronic device 101) that generates a path rulethrough user log collection of the electronic device according tovarious embodiments. Referring to FIG. 16 , the electronic device maygenerate a path rule that does not exist in an existing rule database bypattern analysis.

In operation 1610, the electronic device (e.g., the processor 120) maycollect application state logs in which an input means is classified.

According to an embodiment, when the electronic device executes at leastone operation of an application via a user's voice input using amicrophone, via a user's touch input using a touch screen, or via amixed input of user's voice and touch, the electronic device mayclassify the corresponding input means.

According to an embodiment, a single screen displayed according to oneoperation executed by the electronic device or the execution of the oneoperation may be referred to as a state, and recording of operationsexecuted by the electronic device or recording of screens displayed tocorrespond to the executed operations may be referred to as a state log.

According to an embodiment, the electronic device may collect contextinformation together with the state logs. For example, the contextinformation may be various, such as a current date, a current time, alocation where the electronic device is located, and/or scheduleinformation stored in the electronic device.

In operation 1620, the electronic device (e.g., the processor 120) maydetermine whether there is a state sequence log corresponding to thecollected state logs in a rule database. In operation 1620, when it isdetermined that there is the state sequence log corresponding to thecollected state logs in the rule database, the electronic device mayperform operation 1640, and otherwise, may perform operation 1630.

According to an embodiment, the electronic device may determine whetherthere is a portion where the application state log corresponds to thestate sequence log, by comparing the application state log with thestate sequence log stored in the existing rule database through the rulemapper process. For example, when there is the portion where theapplication state log corresponds to the state sequence log, theelectronic device may perform operation 1640, and otherwise, may performoperation 1630.

In operation 1630, the electronic device (e.g., processor 120) maygenerate a new path rule through pattern analysis of the collected statelogs.

According to an embodiment, the electronic device or the processor(e.g., the processor 120) of the electronic device may include a rulepattern finder (not shown), and the rule pattern finder may analyze therepeatedly used state log through pattern analysis of the collectedstate logs to extract a new path rule.

For example, when the electronic device executes a gallery applicationaccording to a user's touch input or the like and a state sequence logof attaching photos to a note application does not exist in the existingpath rule database, or when the rule pattern finder analyzes the statelogs collected according to the operation of the electronic device andthe state log is performed a designated reference number of times ormore, it may be confirmed that a user has performed the above-describedfunction repeatedly, and the path rule including the state sequence maybe newly acquired. For example, the electronic device may store thenewly acquired path rule in the rule database and may update the ruledatabase. For example, the electronic device may execute a confirmationprocess for the newly acquired path rule, and then may store the resultin the rule database.

In operation 1640, the electronic device (e.g., the processor 120) maydetermine the path rule according to the application state log throughthe rule mapper processor.

According to an embodiment, the electronic device may confirm to whichportion the portion where the application state log corresponds to thestate sequence log corresponds, by comparing the application state logwith the state sequence log stored in the existing rule database throughthe rule mapper process, and may determine the path rule correspondingto the application state log from the state sequence log.

According to an embodiment, the path rule may include information of theinput means so that it can be known through which interface a userperforms an input.

According to an embodiment, when the path rule is an utterance pathrule, the electronic device (e.g., the processor 120) may store the pathrule in an utterance path rule table in operation 1650. For example,when the information of the input means included in the path rule is avoice input using a microphone, the electronic device may determine thatthe path rule is the utterance path rule. For example, the electronicdevice may store the context information in the utterance path ruletable together with the path rule.

According to an embodiment, when the path rule is a touch path rule, theelectronic device (e.g., the processor 120) may store the path rule in atouch path rule table in operation 1660. For example, when theinformation of the input means included in the path rule is a touchinput using a touch screen, the electronic device may determine that thepath rule is the touch path rule. For example, the electronic device maystore the context information in the touch path rule table together withthe path rule.

According to an embodiment, when the path rule is a touch and utterancepath rule, the electronic device (e.g., the processor 120) may store thepath rule in a touch and utterance path rule table in operation 1670.For example, when the information of the input means included in thepath rule is the voice input using the microphone and the touch inputusing the touch screen, the electronic device may determine that thepath rule is the touch and utterance path rule. For example, theelectronic device may store the context information in the touch pathrule table together with the path rule.

In operation 1680, the electronic device (e.g., the processor 120) maycombine the path rule of the utterance path rule table, the path rule ofthe touch path rule table, and the touch and utterance path rule of thetouch and utterance path rule table, and may store the combined resultin an utterance/touch combined path rule table. For example, theelectronic device may further store the context information together.For example, the electronic device may make a database together with thenumber of times (frequency) the path rule is performed by a user, andmay store the database. For example, the electronic device may alsostore information indicating whether the path rule has been performedvia the touch input or the voice input, or whether the path rule hasbeen performed via a combination of the two inputs. For example, theelectronic device may also store information indicating which the userinput (referred to as input means or interface) has been applied in eachstate of the path rule so as to be distinguished.

In operation 1690, the electronic device (e.g., the processor 120) mayassign a priority for providing a hint to each path rule included in theutterance/touch combined path rule table according to a designatedcondition, and may store the result in a hint path rule table inoperation 1690. For example, the designated condition may be a frequencyof use of each path rule.

Meanwhile, in the above-described embodiment of FIG. 16 , it has beendescribed that the electronic device generates the path rule throughuser log collection of the electronic device. However, according toanother embodiment, the electronic device (e.g., the server 106) mayreceive operations executed by an external electronic device accordingto the user input or recording of the operations from the externalelectronic device via a wireless communication circuit, and may beoperated in a method similar to the above-described operation(corresponding method). For example, the electronic device may executeoperations of collecting the state logs of the application in which aninput means of the external electronic device is classified, determiningwhether there is the state sequence log corresponding to the collectedstate logs in the rule database, generating a new path through state logpattern analysis, determining the path rule through the log mapperprocessor, storing the path rules in the utterance path rule table, thetouch path rule table, and/or the touch and utterance path rule table,combining the path rules to store the combined result in theutterance/touch combined path rule table, and assigning the priority forproviding the hint to each path rule included in the utterance/touchcombined path rule table to store the result in the hint path ruletable. For example, the electronic device may generate the path rulethrough state log collection of the application for each of a pluralityof external devices (users), thereby having the hint path rule table foreach of the plurality of external devices (users).

According to still another embodiment, when the electronic device (e.g.,the server 106) receives operations executed by the external electronicdevice according to the user input or recording of the operations fromthe external electronic device via the wireless communication circuitand is operated in a method similar to the above-described operation(corresponding method), an operation of detecting a new path rulethrough the above-described state log pattern analysis of the externalelectronic device may not be limited to the corresponding singleexternal electronic device of the received operation or operations. Forexample, through the state log pattern analysis, the new path rule maybe generated through the state log patterns of all external electronicdevices associated with the electronic device. For example, theelectronic device may anonymize and store the application state logs ofthe external electronic devices, and may acquire the new path rule thatis not included in the existing state sequence log in the anonymizedstate logs through pattern analysis. The new path rule thus acquired inthis manner may be newly added to the rule database of the electronicdevice or the external electronic device and may managed.

FIG. 17 is a diagram illustrating a hint table corresponding to a pathrule of an electronic device (e.g., the electronic device 101) accordingto various embodiments. FIGS. 18 to 20 are block diagrams illustrating asystem for providing a hint.

According to an embodiment, the electronic device may classify at leastone hint corresponding to path rules generated according to any oneembodiment of the above-described embodiments into a function guide hinttable 1701, a user usage-based hint table 1703, or a context informationhint table 1705 to store the result, and may allow hints included in thefunction guide hint table 1701, the user usage-based hint table 1703,and the context information hint table 1705 to be included in a hinttable 1710.

According to an embodiment, when the generated path rule is a path ruleof a new function of the electronic device and/or a function that isfrequently used in the electronic device by other people, the electronicdevice may store the hint corresponding to the generated path rule inthe function guide hint table 1701. Thus, the hints included in thefunction guide hint table 1701 may serve to introduce the new functionof the electronic device to the user or to introduce the function whichis frequently used in the electronic device by other people, and theuser may provide a function that is expected to be used among not usedfunctions in the functions of the electronic device, as a hint or a hintlist. For example, the electronic device may prevent a hint that hasbeen used by a user even once or a hint corresponding to the function inuse among the hints stored in the function guide hint table 1701 frombeing provided.

For example, the hints (referred to as function guide hints) stored inthe function guide hint table 1701 may be as shown in Table 2 below.

TABLE 2 Applications Hint examples Gallery application Make a collageSecure folder application Keep important data in a secure folder Themeapplication Would you like to switch it to a new theme? Telephoneapplication You can record calls automatically Message application Wouldyou like to set message background screen? Internet application You canchange basic search engine S note application Take notes easily with Snote Camera application You can store location information together inphotos/videos S health application Measure your heart rate with S Health

For example, the function guide hint is intended to encourage a user touse the function, so it may be effective to initially expose short andsimple hints.

According to the above-described embodiment, although it has beendescribed that the hints of Table 2 are the function guide hints storedin the function guide hint table 1701, according to another embodiment,at least some of the function guide hints of Table 2 may be hintsincluded in the user usage-based hint table 1703 and/or the contextinformation hint table 1705. For example, the hints such as “Make acollage”, “Keep important data in a secure folder”, and/or “Would youlike to switch it to a new theme?” may be hints included in the functionguide hint table 1701, the user usage-based hint table 1703, and/or thecontext information hint table 1705.

Referring to FIG. 18 , when a first electronic device 1801 (e.g., theelectronic device 101 makes a request to provide a hint, a hintproviding module 1807 in a suggestion manager 1805 of a secondelectronic device (e.g., a server) 1803 may receive the correspondingrequest, and may make a request to generate and/or provide a hint from afunction guide hint generation module 1809. The function guide hintgeneration module 1809 may be synchronized with a third electronicdevice (e.g., an external hint suggestion server) 1811 everypredetermined period to update the hint list. For example, the functionguide hint generation module 1809 may generate the hint list in the formof the function guide hint table 1701. When the function guide hintgeneration module 1809 returns the generated hint to the hint providingmodule 1807, the hint providing module 1807 may array the hints to bedisplayed to the user according to a designated condition and mayprovide the arrayed hints to the first electronic device 1801. Thedesignated condition may be a priority determined based on the contextinformation of the electronic device and/or a frequency of use of thehint.

In FIG. 18 described above, it has been described that the hint list isgenerated in the form of the function guide hint table according to themutual operation of the first electronic device 1801, the secondelectronic device 1803, and the third electronic device 1811 and thehints to be displayed to the user are arrayed and provided. However,according to another embodiment, the first electronic device 1801 maygenerate the hint list in the form of the function guide hint table, andmay arrayed and provide hints to be displayed to the user.

According to an embodiment, when the generated path rule is a path rulecorresponding to a hint exposed in a user's specific context, theelectronic device may store the hint corresponding to the generated pathrule in the context information hint table 1705. For example, thecontext information hint table 1705 may include the hints correspondingto the path rules generated in consideration of time, place, and/orcontext information. For example, when a predefined condition issatisfied based on time, place, occasion (TPO) information, theelectronic device may provide a recommended hint or hint list inconsideration of the TOP information, that is, a time during which theelectronic device is used, a place where the electronic device is use,and/or context information when the electronic device is used. Forexample, the electronic device may select at least one hint suitable forthe predefined condition based on TPOA information in a preset hintpool, and may provide the selected hint to the user.

For example, the hints (referred to as context information hints) storedin the context information hint table 1705 may be as shown in Table 3below.

TABLE 3 Conditions Context information hint examples Next event within 2hours Let me know the next schedule Next event location and (1 hour Giveme directions to the next before or on board) appointment place Nextevent location and (if not 1 Call taxi to the next hour before or duringboarding) appointment place Next event location and Search near the next(event schedule is before quitting appointment place time, that is, notduring daily work) Quitting time and on board Give me directions to homeNot quitting time and on board Call taxi to home Attendance time and onboard Give me directions to company Not attendance time and on boardCall taxi to company After wake-up What about the weather today? Afterwake-up Show me today's schedule alarm after wake-up, during the Turnoff all alarms morning (to 12 o'clock), and within one hour Beforebedtime Set obstruction prohibition Before bedtime What about theweather tomorrow? Before bedtime Show me tomorrow's schedule No alarmbefore bedtime and Set alarm during the morning (to 12 o'clock) Beforebedtime and Sunday What about the weather next week? unread e-mail fromVIP Show me mail from VIP New e-mail Show me new mail multiple missedcalls Show me missed call missed calls Call {missed phone number} Newmessage Show me new message Voice mail Show me voice mail

For example, house recognition, company recognition, whether to board avehicle, wake-up time, bedtime, attendance time, and quitting time ofTable 3 may use values obtained through a personal manager 1919 of FIG.19 to be described later, and initial values shown in Table 4 may beused for the time.

TABLE 4 Wake-up time Bedtime Attendance time Quitting time 7:00 22:0008:00~09:00 18:00~19:00

For example, when providing the context information hint, the electronicdevice may designate a priority of the context information of the userwhile providing an appropriate command hint according to current contextinformation, and may provide preferentially provide the hints having ahigher priority. For example, the electronic device may assign apriority to each context information as shown in Table 5 below.

TABLE 5 Context information types Examples Priorities User occasionEvent or the like predicted by user's 1 schedule and user's behaviorOccasion Wake-up, before bed, attendance time, 2 end time, travel,overseas etc. Place In house, outside house, inside company, 3 outsidecompany, inside car, outside car, overseas Time Morning (6 to 11), lunch(11 to 13), 4 afternoon (13 to 18), evening (18 to 23), night (23 to 6)etc. In application View contents, selected plurality of items 5

Referring to FIG. 19 , when a first electronic device 1901 (e.g., theelectronic device 101) makes a request to provide a hint, a hintproviding module 1907 in a suggestion manager 1905 of a secondelectronic device 1903 may receive the request, and may make a requestto generate and/or provide a hint from a context information hintgeneration module 1909. The context information hint generation module1909 may receive TPO information using a module in a context informationmanager 1915 or a personal manager 1919 using a condition checkingmodule 1913. For example, the context information manager 1915 mayinclude a location proximity module 1917 that determines the proximityof a place, and the personal manager 1919 may include a placeinformation module 1921 that stores location information of the placeand an inference engine 1923 that infers the location information andcontext information related to the place. For example, the placeinformation module 1921 may store an address of a house and/or a companyinput by the user. For example, the inference engine 1923 may infer alocation of a user's home and/or company or location-related contextinformation (e.g., during commuting, traveling, driving, etc.) from theinformation input by the user and/or the information collected using oneor more sensors of the electronic device.

According to an embodiment, the context information hint generationmodule 1909 may set a function guide hint generation module 1911 in thesuggestion manager 1905 using the received TPO information and thecondition checking module 1913 may check the function guide hintgeneration module 1911, so that the context information hint generationmodule 1909 may generate the hint suitable for the occasion.

According to another embodiment, the context information hint generationmodule 1909 may set a user usage-based hint selection module 1912 in thesuggestion manager 1905 using the received TPO information and thecondition checking module 1913 may check the user use-base hintselection module 1912, so that the user usage-base hint selection module1912 may select the usage-based hint of the user. For example, assumingthat a user of the first electronic device 1901 frequently uses a voicecommand of “Play music” at home, when it is determined that the locationof the electronic device is home, the user usage-based hint selectionmodule 1912 may select “Play music” in which the reuse frequency isconsidered, as the usage-based hint of the user.

When the hints generated according to the above-described embodimentsare returned to the hint providing module 1907, the hint providingmodule 1907 may array and/or select the hint to be provided according tointernal rules, and may transmit the selected hint to the firstelectronic device 1901.

In FIG. 19 described above, it has been described that the contextinformation hint is generated according to the mutual operation of thefirst electronic device 1901 and the second electronic device 1903 andmay array and provide the hints to be displayed to the user. However,according to another embodiment, the first electronic device 1901 maygenerate the context information hint and may array and provide thehints to be displayed to the user.

According to an embodiment, when the generated path rule is a path rulecorresponding to the hint considering a use frequency (or a frequency),the electronic device may store the generated path rule in the userusage-based hint table 1703. For example, the user usage-based hinttable 1703 may include a hint corresponding to a path rule considering areuse frequency of the hint.

According to an embodiment, although not shown, the hint or hint list tobe provided to the user may be managed by a method in which an operatorarbitrarily inputs a manual to an external hint suggestion server (e.g.,the third electronic device 1811) for management, or a method in whichthe external hint suggestion server extracts a popular utterance listand/or hint list which is frequently used in the electronic device(e.g., the first electronic device 1801) to update the external hintsuggestion server.

Referring to FIG. 20 , when the generated path rule is a path rulecorresponding to the hint considering a use frequency (or a frequency),the electronic device (e.g., the electronic device 101) may store thehint corresponding to the generated path rule as the user usage-basedhint. For example, the electronic device may provide a hint forrecommending a function, which is frequently used through a userutterance or a text input, according to the use frequency based on userpersonal TPO information. For example, the electronic device mayrecommend the function frequently used by the user so that, in a case inwhich the user is currently unable to speak or in a long sentence, adesired function may be performed by performing a single touch inputrather than uttering the long sentence.

Referring to FIG. 20 , when a first electronic device 2001 (e.g., theelectronic device 101) makes a request to provide a hint, a hintproviding module 2007 in a suggestion manager 2005 of a secondelectronic device 2003 may receive the request, and may make a requestto generate and/or provide a hint from the user usage-based hintgeneration module 2009. The user usage-based hint generation module 2009may generate a user usage-based hint using a user's dialog logged in auser utterance preference recording module 2013 in a personal manager2011, a path rule ID, a parameter, a use frequency, used time and placeinformation, and the like. When the generated user usage-based hint istransmitted to the hint providing module 2007, the hint providing module2007 may array the hints to be displayed to the user according to theinternal rules, and may provide the selected user usage-based hint tothe first electronic device 2001.

According to an embodiment, the second electronic device 2003 mayrecommend the hint according to a user's usage pattern of the firstelectronic device 2001. For example, when the second electronic device2003 periodically receives a request to inform the weather from thefirst electronic device 2001, context information in which the requestis made and a user input (a user command) of the first electronic device2001 may be simultaneously managed, so that, when the same contextinformation is obtained, a function to be executed by the user may bepredicted to provide a hint. For example, in a case in which theexecuted function is meaningless because it is one-time function or theexecuted function overlap with other automation functions, the functionmay not be stored as a hint.

According to an embodiment, although not shown, the hint or hint list tobe provided to the user may be managed by a method in which an operatorarbitrarily inputs a manual to an external hint suggestion server (e.g.,the third electronic device 1811) for management, or a method in whichthe external hint suggestion server extracts a popular utterance listand/or hint list which is frequently used in the electronic device(e.g., the first electronic device 1801) to update the external hintsuggestion server.

FIG. 21 is a diagram illustrating a system that utilizes a private cloudto provide a personalized hint according to various embodiments.

Referring to FIG. 21 , a hint table 2110 (e.g., the hint table 1710)stored in a server 2111 may be stored in a private cloud 2121 beforebeing provided to an electronic device 2131 and may be synchronized withthe electronic device 2131. For example, the server 2111 may generateand/or update the hint table 2110 by executing the hint generationoperation described above and may generate and/or update the hint table2120 corresponding to the hint table 2110, so that the hint table 2120may be stored in the corresponding private cloud 2121 of the electronicdevice 2131, which is a storage space of a personalized hint. Forexample, the hint table 2110 may include all of hints included in afunction guide hint table 2101 (e.g., the function guide hint table1701), a user usage-based hint table 2103 (e.g., the user usage-basedhint table 1703), and a context information hint table 2105 (the relatedhint table 1705). The electronic device 2131 may be provided with a hintor a hint list in conjunction with the context information, personalprofile information, and the hint table 2120 of the private cloud 2121.

FIG. 22 is a block diagram illustrating a system for providing a hintaccording to various embodiments.

Referring to FIG. 22 , the system for providing a hint may include anelectronic device 2201 and a server 2231.

According to an embodiment, the electronic device 2201 may be developedand operated based on an OS framework 2205.

According to an embodiment, the electronic device 2201 may include a bigdata collection client 2210, and the big data collection client 2210 mayinclude a log collector 2211, an inference engine 2213, a contextinformation manager 2215, a personal manager 2216, a suggestion manager2217, a content providing application programming interface (API) 2219.

According to an embodiment, the log collector 2211 may collect logs(also referred to as log data) of the electronic device 2201. Forexample, the log collector 2211 may collect state logs of the electronicdevice 2201 according to a user input at the electronic device 2201. Forexample, the user input may be various, such as a voice input, a touchinput, a user manipulation of buttons (or keys), and/or an input using aremote controller.

According to an embodiment, the inference engine 2213 may infer contextinformation of a user and/or the electronic device 2203 from informationinput by the user and/or information collected using one or more sensorsof the electronic device 2201. For example, the inference engine 2213may infer the context information of the user and/or the electronicdevice 2201 based on the logs of the electronic device 2201 collectedthrough the log collector 2211. For example, the inference engine 2213may infer a location of a user's home and/or company or location-relatedcontext information (e.g., during commuting, traveling, driving, etc.).

According to an embodiment, a context information manager 2215 maycollect context information of the electronic device 2201. For example,the context information manager 2215 may collect the context informationof the electronic device 2201 inferred using the inference engine 2213.For example, the context information may include general contextinformation, user context information, and/or device contextinformation.

For example, the general context information may include general contextinformation of the electronic device 2201. The general contextinformation may be confirmed through an internal algorithm by receivingdata through a sensor hub of a device platform or the like. For example,the general context information may include information about a currenttime and space. The information about the current time and space mayinclude, for example, information about a current time and a currentlocation of the electronic device 2201. The current time may beconfirmed through time on the electronic device 2201, and theinformation about the current location may be confirmed through a globalpositioning system (GPS). By way of another example, the general contextinformation may include information about physical movement. Theinformation about physical movement may include, for example,information about walking, running, driving, and the like. Theinformation about physical movement may be confirmed through a motionsensor. The information about driving may be confirmed through themotion sensor, and boarding and parking may be confirmed by sensing aBluetooth connection in a vehicle. By way of still another example, thegeneral context information may include user activity information. Theuser activity information may include, for example, information aboutcommuting, shopping, traveling, and the like. The user activityinformation may be confirmed using information about a user or a placewhich is registered in a database by an application.

For example, the user context information may include information aboutthe user. For example, the user context information may includeinformation about emotional states of the user. The information aboutthe emotional states may include, for example, information about user'shappiness, sadness, anger, and the like. By way of another example, theuser context information may include information about the current stateof the user. The information about the current state may includeinformation about, for example, interest, intention, etc. (e.g.,shopping).

For example, the device context information may include informationabout the state of the electronic device 2201. For example, the devicecontext information may include information about path rules executed bythe electronic device 2201. By way of another example, the devicecontext information may include information about a battery. Theinformation on the battery may be confirmed through, for example, thestate of charge and discharge of the battery. By way of still anotherexample, the device context information may include information about aconnected device and network. The information about the connected devicemay be confirmed, for example, via a communication interface to whichthe device is connected.

According to an embodiment, the context information manager 2215 maytransmit the context information to the suggestion manager 2217.

According to an embodiment, the personal manager 2216 may manage user'spersonal information using the electronic device 2201. For example, thepersonal manager 2216 may collect usage information and performanceresults of the electronic device 2201 to manage the user's personalinformation. For example, the personal manager 2216 may collect theuser's personal information based on the context information of theelectronic device 2201 inferred using the inference engine 2213. Forexample, the personal information of the user using the electronicdevice 2201 may include information about operation result informationof the application executed by the electronic device 2201 and/orinformation about the current state of the electronic device 2201.

According to an embodiment, the personal manager 2216 may transmit thepersonal information of the user using the electronic device 2201 to thesuggestion manager 2217.

According to an embodiment, the personal manager 2216 may transmit theuser's personal information to the server 2231. For example, thepersonal manager 2216 may periodically transmit the received andaccumulated user's personal information to the server 2231.

According to an embodiment, the suggestion manager 2217 may predict auser's intention and may recommend a hint to the user to perform anoperation of the electronic device. For example, the suggestion manager2217 may recommend the hint to the user in consideration of the currentstate of the user (e.g., time, place, occasion, used application). Forexample, the suggestion manager 2217 may recommend the hint forexecuting the operation of the electronic device 2201 using the contextinformation of the electronic device 2201 received via the contextinformation manager 2215, the user's personal information of theelectronic device 2201 received via the personal manager 2216, and/orthe user context information inferred by the inference engine 2213.

According to an embodiment, the electronic device 2201 may include afirst application 2221, a second application 2223, a third application2225, and/or a fourth application 2227.

According to an embodiment, a content providing application programminginterface (API) 2219 may transmit information collected and generated bythe big data collection client 2210, to the first, second, third, and/orfourth applications 2221, 2223, 2225, and 2227 of the electronic device2201.

According to an embodiment, the server 2231 may communicate with theelectronic device 2201 via a fronted API 2233 of the server 2231 and maybe requested to provide a hint from the electronic device 2201.

According to an embodiment, the server 2231 may include an analysismodule 2235 (also referred to as an intelligent module), a servicemanagement module 2245, and a log/context information management module2249.

According to an embodiment, the analysis module 2235 may include ageneral profile engine 2237, a feedback-based user profile engine 2239,a domain specific profile engine 2241, and a priority platform API 2243.

According to an embodiment, the general profile engine 2237 may generatea general profile based on information received from the electronicdevice 2201.

According to an embodiment, the feedback-based user profile engine 2239may generate a feedback-based profile based on information aboutfeedback received from the electronic device 2201. For example, thefeedback-based user profile engine 2239 may generate the feedback-basedprofile using a reinforcement learning approach.

According to an embodiment, the domain specific profile engine 2241 maygenerate a profile for each domain by classifying the informationreceived from the electronic device 2201 for each domain. The domain maybe various, such as health, intention extraction, and the like.

According to an embodiment, the priority platform API 2243 may generatea priority condition for determining a hint to be provided to theelectronic device 2201 based on time, location, and/or occasion. Forexample, the priority platform API 2243 may be referred to as aTPO-based analyzer.

According to an embodiment, the service management module 2245 mayinclude a user/device management manager 2247. For example, theuser/device management manager 2247 may manage the device contextinformation of the electronic device 2201. For example, the devicecontext information may include information about the battery of theelectronic device 2201, information about a CPU of the electronic device2201, and/or information about a device and network connected to theelectronic device 2201.

According to an embodiment, the log/context information managementmodule 2249 may include a log collection manager 2251 and a contextinformation manager 2253. For example, the log collection manager 2251may collect and store logs of the electronic device 2201. For example,the context information manager 2253 may store context information atthe time of log collection of the electronic device 2201.

According to an embodiment, the server 2231 may further include aserver-to-server API 2255, and may communicate with a 3rd partyapplication 2261 via the server-to-server API 2255. For example, theserver 2231 may receive information about the electronic device 2201(e.g., information used by the electronic device 2201 in the 3rd partyapplication 2261) from the 3rd party application 2261. For example, the3rd party application 2261 may be various, such as “viv,” a serviceserver, an Internet of things (IOT) cloud, or the like.

FIG. 23 is a flowchart illustrating an operation for hint generation ofa server according to various embodiments.

In operation 2301, the server may collect data of the electronic device.

According to an embodiment, the data of the electronic device mayinclude device information, application usage information of theelectronic device, and/or service usage information. For example, thedevice information may include information such as the location of theelectronic device, the movement of the electronic device, a Bluetoothconnection of the electronic device, a screen-on of the electronicdevice, and/or setting change of the electronic device. For example, theapplication usage information of the electronic device may include aforeground application, an application setting change, a contact use, acommand, an application touch, a URL, a search keyword, music, and thelike. For example, the service usage information may include usage logsof services such as the use of an IOT device and Samsung Pay.

In operation 2303, the server may generate a profile of the electronicdevice.

According to an embodiment, the server may aggregate the log data of theelectronic device over a long period of time and may generate theprofile of the electronic device based on the aggregated log data.

According to an embodiment, the profile may include a location profile,a behavior profile, and/or a user profile. For example, the locationprofile may include a home, a workplace, a car, a frequently visitedlocation, and/or a user input location. For example, the behaviorprofile may include a sleeping time, and/or commuting time. For example,the user profile may include a user's interest category and keywords,demographics, and/or a service specific user profile.

In operation 2305, the server may generate the context information ofthe electronic device.

According to an embodiment, the server may combine the generated profilewith the log data of the electronic device to generate the contextinformation of the electronic device.

According to an embodiment, the context information may include placecontext information and/or context information. For example, the placecontext information may include information such as inside . . . ,outside . . . , in . . . , nearby . . . , an exceptional path, everydaylife, an unusual place, domestic, overseas, and/or travel. For example,the context information may include information such as before sleep,after sleep, before commuting, after commuting, during commuting, beforedriving, after driving, during driving, and/or weather.

In operation 2307, the server may generate a priority condition fordetermining a hint to be provided to the electronic device.

According to an embodiment, the server may combine the contextinformation and the log data to generate the priority condition.

According to an embodiment, the priority condition may include a generalpriority condition, a service specific priority condition, and/or asuggestion priority condition. For example, the general prioritycondition may include application preference, contact preference,setting preference, location preference, music preference, URL, and/orsearch keywords. For example, the service specific priority conditionmay include hint command preference, IOT preference, and/or servicepreference. For example, the suggestion priority condition may includecontent based and/or collaborative filtering.

FIG. 24 is a graph illustrating a type of a hint to be provided inconsideration of a use time of hint providing of an electronic deviceaccording to various embodiments and a hint exposure frequency thereof.

Referring to FIG. 24 , in the beginning of the function of providing thehint according to the above-described embodiments, since personal dataof a user logged in the electronic device is small, a function guidehint 2401 (e.g., the hint included in the function guide hint table1701) and a context information hint 2403 (e.g., the hint included inthe context information hint table 1705) may be mainly exposed. However,as the user's use of the above-mentioned functions increases, aprobability of recommending a user usage-based hint 2405 (e.g., the hintincluded in the user usage-based hint table 1703) increases, therebyproviding a personalized hint. However, even if the ratio of the userusage-based hints increases, it may be desirable to maintain theexposure rate of the function guide hint 2401 at a predetermined levelor more. This can be utilized to induce to use the correspondingfunction by introducing the function which the user has not used amongthe less exposed hints.

According to an embodiment, assuming that the use of any particular hintby the user is repeated, the user usage-based hint may be displayed mostpreferentially. The priority of the hint which has been shown to theuser several times but has not been selected may be reduced. Inaddition, once the user has achieved the purpose, the hint may be keptfrom appearing for a while.

According to an embodiment, in the operation in which the electronicdevice provides the hint to the user, the electronic device may providethe hints with different priorities to the user according to the type ofeach of the hints. For example, exposure priorities of the hints may bedetermined in the order of personalized hints, context informationhints, and application hints, and the total number of hints to beexposed may be limited. In the case of the personalized hints, the hintsmay be generated through rotation according to the priority for eachcontext information. In the case of the context information hints, whenthere is no the personalized hint, the hints may be displayed accordingto the priorities.

According to an embodiment, when the user usage-based hint issignificantly short, the recommendation may be excluded.

According to an embodiment, the above-described state sequence log maybe interlocked with a user's account on the server. For example, theuser's account may have multiple electronic device IDs, and the same ordifferent device types may exist.

According to various embodiments, the hints of the electronic device ofthe user may be shared with other electronic devices of the user. Forexample, when the electronic device performs an operation in conjunctionwith another electronic device (e.g., a video of a terminal is executedon a TV) or when the electronic device operates independently (e.g., avideo search on a TV), the hint may be provided from another electronicdevice (e.g., a mobile device).

FIG. 25 is a diagram illustrating an integrated intelligent systemaccording to various embodiments of the present disclosure.

Referring to FIG. 25 , an integrated intelligent system 251 may includea user terminal 2500 (e.g., the electronic device 101), an intelligentserver 2600 (e.g., the server 106), a personalization information server2700 (e.g., the server 106), or a suggestion server 2800 (e.g., theserver 106).

The user terminal 2500 may provide services to a user through apps (orapplication programs or also referred to as applications (e.g., an alarmapp, a message app, a photo (gallery) app, etc.) stored in the userterminal 2500. For example, the user terminal 2500 may execute andoperate other apps through an intelligent app (or a voice recognitionapp) stored in the user terminal 2500. A user input for executing theoperation of the other application through the intelligent app of theuser terminal 2500 may be received. The user input may be received via,for example, a physical button, a touchpad, a voice input, a remoteinput, or the like. According to an embodiment, the user terminal 2500may correspond to various terminal devices (or electronic devices) thatcan be connected to the Internet, such as a mobile phone, a smart phone,a personal digital assistant (PDA), or a notebook computer.

According to an embodiment, the user terminal 2500 may receive a user'sutterance as the user input. The user terminal 2500 may receive theuser's utterance and may generate a command to operate the app based onthe user's utterance. Accordingly, the user terminal 2500 may operatethe application using the command.

The intelligent server 2600 may receive a user voice input from the userterminal 2500 via a communication network and may convert the receiveduser voice input into text data. According to another embodiment, theintelligent server 2600 may generate (or select) a path rule based onthe text data. The path rule may include information about an action (oran operation) for performing the function of the app or informationabout a parameter for executing the operation. In addition, the pathrule may also include a sequence of the operations of the app. The userterminal 2500 may receive the path rule, may select an app according tothe path rule, and may execute the operation included in the path rulein the selected app.

For example, the user terminal 2500 may perform the above-describedoperation and may display a screen corresponding to the state of theuser terminal 2500 that has performed the operation, on a display. Byway of another example, the user terminal 2500 may execute the operationand may not display the result of performing the operation on thedisplay. The user terminal 2500 may, for example, execute a plurality ofoperations and may display some of the results of the operations on thedisplay. The user terminal 2500 may, for example, display the result ofperforming the last operation on the display. By way of still anotherexample, the user terminal 2500 may receive the user input and maydisplay the result of performing the above-described operation on thedisplay.

The personalization information server 2700 may include a database inwhich user information is stored. For example, the personalizationinformation server 2700 may receive user information (e.g., contextinformation, application execution, etc.) from the user terminal 2500and may store the received user information in the database. Theintelligent server 2600 may be used when receiving the user informationfrom the personalization information server 2700 through a communicationnetwork and generating the path rule for the user input. According to anembodiment, the user terminal 2500 may receive the user information fromthe personalization information server 2700 through the communicationnetwork and may use the received user information as information formanaging the database.

The suggestion server 2800 may include a database in which informationabout an introduction of a function or an application or informationabout a function to be provided is stored in the terminal. For example,the suggestion server 2800 may include a database for a function ofreceiving the user information of the user terminal 2500 from thepersonalization information server 2700 so that the user may use theuser information. The user terminal 2500 may receive information aboutthe function to be provided from the suggestion server 2800 through thecommunication network and may provide the received information to theuser.

FIG. 26 is a block diagram illustrating a user terminal of an integratedintelligent system 251 according to an embodiment of the presentdisclosure.

Referring to FIG. 26 , the user terminal 2500 may include an inputmodule 2510 (e.g., the input device 250), a display 2520 (e.g., thedisplay 260), a speaker 2530 (e.g., the speaker 282), a memory 2540(e.g., the memory 230), or a processor 2550 (e.g., the processor 210).The user terminal 2500 may further include a housing, and theconfigurations of the user terminal 2500 may be placed inside thehousing or on the housing.

According to an embodiment, the input module 2510 may receive the userinput from the user. For example, the input module 2510 may receive theuser input from a connected external device (e.g., a keyboard, aheadset). By way of another example, the input module 2510 may include atouch screen (e.g., a touch screen display) coupled with the display2520. By way of still another example, the input module 2510 may includea hardware key (e.g., 2512 in FIG. 27 ) (or a physical key) positionedin the user terminal 2500 (or the housing of the user terminal 2500).

According to an embodiment, the input module 2510 may include amicrophone (e.g., 2513 a or 2513 b in FIG. 27 ) capable of receiving auser's utterance as a voice signal. For example, the input module 2510may include a speech input system and may receive the user's utteranceas the voice signal via the speech input system.

According to an embodiment, the display 2520 may display images, videos,and/or execution screens of the applications. For example, the display2520 may display a graphical user interface (GUI) of the app.

According to an embodiment, the speaker 2530 may output a voice signal.For example, the speaker 2530 may output the voice signal generatedinside the user terminal 2500 to the outside.

According to an embodiment, the memory 2540 may store a plurality ofapps 2541 and 2543. The plurality of apps 2541 and 2543 stored in thememory 2540 may be selected, executed, and operated according to theuser input.

According to an embodiment, the memory 2540 may include a databasecapable of storing information utilized to recognize the user input. Forexample, the memory 2540 may include a log database capable of storinglog information. By way of another example, the memory 2540 may includea persona database capable of storing the user information.

According to an embodiment, the memory 2540 may store the plurality ofapps 2541 and 2543, and the plurality of apps 2541 and 2543 may beloaded and operated. For example, the plurality of apps 2541 and 2543stored in the memory 2540 may be loaded and operated by an executionmanager module 2553 of the processor 2550. The plurality of apps 2541and 2543 may include execution services 2541 a and 2543 a or a pluralityof operations (or unit operations) 2541 b and 2543 b which performfunctions. The execution services 2541 a and 2543 a may be generated bythe execution manager module 2553 of processor 2550 to execute theplurality of operations 2541 b and 2543 b.

According to an embodiment, when the operations 2541 b and 2543 b of theapps 2541 and 2543 are executed, an execution state screen according tothe execution of the operations 2541 b and 2543 b may be displayed onthe display 2520. The execution state screen may be, for example, ascreen in which the operations 2541 b and 2543 b are completed. By wayof another example, the execution state screen may be a screen forpartial landing (for example, when parameters utilized for theoperations 2541 b and 2543 b are not inputted) in which the execution ofthe operations 2541 b and 2543 b is stopped.

According to an embodiment, the execution services 2541 a and 2543 a mayexecute the operations 2541 b and 2543 b according to the path rule. Forexample, the execution services 2541 a and 2543 a may be activated bythe execution manager module 2553, may receive an execution requestaccording to the path rule from the execution manager module 2553, andmay execute the operations of the apps 2541, 2543. When the execution ofthe operations 2541 b and 2543 b are completed, the execution services2541 a and 2543 a may transmit completion information to the executionmanager module 2553.

According to an embodiment, when the plurality of operations 2541 b and2543 b are executed in the apps 2541 and 2543, the plurality ofoperations 2541 b and 2543 b may be executed sequentially. The executionservices 2541 a and 2543 a may open the next operation (operation 2)when the execution of one operation (operation 1) is completed, and maytransmit completion information to the execution manager module 2553.Here, opening an arbitrary operation can be understood as transitioningthe arbitrary operation to an executable state or preparing for theexecution of the arbitrary operation. In other words, when the arbitraryoperation is not opened, the corresponding operation cannot be executed.The execution manager module 2553 may transmit an execution request forthe next operations 2541 b and 2543 b to the execution service (e.g.,operation 2) when the completion information is received. According toan embodiment, when the plurality of apps 2541 and 2543 are executed,the plurality of apps 2541 and 2543 may be sequentially executed. Forexample, when the execution of the last operation of the firstapplication 2541 is completed and corresponding completion informationis received, the execution manager module 2553 may transmit a request toexecute the first operation of the second app 2543 to the executionservice 2543 a.

According to an embodiment, when the plurality of operations 2541 b and2543 b are executed in the apps 2541 and 2543, a result screen accordingto the execution of each of the plurality of executed operations 2541 band 2543 b may be displayed on the display 2520. According to anembodiment, a portion of a plurality of result screens resulting fromthe execution of the plurality of executed operations 2541 b and 2543 bmay be displayed on the display 2520.

According to an embodiment, the memory 2540 may store intelligent apps(e.g., voice recognition apps) interlocked with an intelligent agent2551. The app interlocked with the intelligent agent 2551 may receiveand process a user's utterance as a voice signal. According to anembodiment, the application interlocked with the intelligent agent 2551may be operated by a specific input (e.g., an input via a hardware key,an input via a touch screen, a specific voice input, or the like) thatis input via the input module 2510.

According to an embodiment, the processor 2550 may control the overalloperation of the user terminal 2500. For example, the processor 2550 maycontrol the input module 2510 to receive the user input. The processor2550 may control the display 2520 to display an image. The processor2550 may control the speaker 2530 to output the voice signal. Theprocessor 2550 may control the memory 2540 to retrieve or store relevantinformation.

According to an embodiment, the processor 2550 may include theintelligent agent 2551, the execution manager module 2553, or anintelligent service module 2555. In an embodiment, the processor 2550may execute commands stored in the memory 2540 to drive the intelligentagent 2551, the execution manager module 2553, or the intelligentservice module 2555. The various modules referred to in the variousembodiments of the present disclosure may be implemented in hardware orsoftware. Operations performed by the intelligent agent 2551, theexecution manager module 2553, or the intelligent service module 2555 invarious embodiments of the present disclosure may be understood asoperations performed by the processor 2550.

According to an embodiment, the intelligent agent 2551 may generate acommand to operate the app based on the voice signal received as theuser input. According to an embodiment, the execution manager module2553 may receive the generated command from the intelligent agent 2551,and may select, execute, and operate the apps 2541 and 2543 stored inthe memory 2540. According to an embodiment, the intelligent servicemodule 2555 may be used to manage the user information and process theuser input.

The intelligent agent 2551 may transmit the user input received via theinput module 2510 to the intelligent server 2600 for processing.

According to an embodiment, the intelligent agent 2551 may pre-processthe user input before transmitting the user input to the intelligentserver 2600. According to an embodiment, the intelligent agent 2551 mayinclude an adaptive echo canceller (AEC) module, a noise suppression(NS) module, an end-point detection (EPD) module, or an automatic gaincontrol (AGC) module in order to pre-process the user input. The AECmodule may remove an echo included in the user input. The NS module maysuppress background noise included in the user input. The EPD module maydetect an end-point of the user voice included in the user input to finda part where the user's voice exists. The AGC module may adjust thevolume of the user input to be suitable for recognizing and processingthe user input. According to an embodiment, the intelligent agent 2551may include all of the above-described preprocessing configurations forthe purpose of performance. However, according to another embodiment,the intelligent agent 2551 may include some of the preprocessingconfigurations to operate at low power.

According to an embodiment, the intelligent agent 2551 may include awake-up recognition module that recognizes a user's call. The wake-uprecognition module may recognize a user's wake-up command via a voicerecognition module, and may activate the intelligent agent 2551 toreceive the user input when the wake-up command is received. Accordingto an embodiment, the wake-up recognition module of the intelligentagent 2551 may be implemented in a low power processor (e.g., aprocessor included in an audio codec). According to an embodiment, theintelligent agent 2551 may be activated according to the user input viaa hardware key. When the intelligent agent 2551 is activated, theintelligent app (e.g., the voice recognition app) interlocked with theintelligent agent 2551 may be executed.

According to an embodiment, the intelligent agent 2551 may include thevoice recognition module for performing the user input. The voicerecognition module may recognize the user input for causing the app toexecute the corresponding operation. For example, the voice recognitionmodule may recognize a limited user (voice) input (e.g., utterance suchas “clicking” that allows a shooting operation to be executed when thecamera app is executed) through which the apps 2541 and 2543 execute anoperation such as the wake-up command. The voice recognition module thatassists the intelligent server 2600 and recognizes the user input mayrecognize and rapidly process the user command that can be processed inthe user terminal 2500. According to an embodiment, the voicerecognition module for executing the user input of the intelligent agent2551 may be implemented in the application processor.

According to an embodiment, the voice recognition module of theintelligent agent 2551 (including the voice recognition module of thewake-up module) may recognize the user input using an algorithm forrecognizing a voice. The algorithm used to recognize the voice may be atleast one of, for example, a hidden Markov model (HMM) algorithm, anartificial neural network (ANN) algorithm, and a dynamic time warping(DTW) algorithm.

According to an embodiment, the intelligent agent 2551 may convert theuser's voice input into text data. According to an embodiment, theintelligent agent 2551 may forward the user's voice to the intelligentserver 2600 to receive the converted text data. Accordingly, theintelligent agent 2551 may display the text data on the display 2520.

According to an embodiment, the intelligent agent 2551 may receive thepath rule transmitted from the intelligent server 2600. According to anembodiment, the intelligent agent 2551 may transmit the path rule to theexecution manager module 2553.

According to an embodiment, the intelligent agent 2551 may transmit anexecution result log according to the path rule received from theintelligent server 2600, to the intelligent service module 2555. Next,the transmitted execution result log may be accumulated and managed inuser's preference information of the persona module 2555 b.

According to an embodiment, the execution manager module 2553 mayreceive the path rule from the intelligent agent 2551 to execute theapps 2541 and 2543, and may cause the apps 2541 and 2543 to execute theoperations 2541 b and 2543 b included in the path rule. For example, theexecution manager module 2553 may transmit command information forexecuting the operations 2541 b and 2543 b to the apps 2541 and 2543,and may receive completion information of the operations 2541 b and 2543b from the apps 2541 and 2543.

According to an embodiment, the execution manager module 2553 maytransmit and receive the command information for executing theoperations 2541 b and 2543 b of the apps 2541 and 2543 between theintelligent agent 2551 and the apps 2541 and 2543. The execution managermodule 2553 may bind the apps 2541 and 2543 to be executed according tothe path rule and may transmit the command information of the operations2541 b and 2543 b included in the path rule to the apps 2541 and 2543.For example, the execution manager module 2553 may sequentially transmitthe operations 2541 b and 2543 b included in the path rule to the apps2541 and 2543 to thereby sequentially execute the operations 2541 b and2543 b of the apps 2541 and 2543 according to the path rule.

According to an embodiment, the execution manager module 2553 may manageexecution states of the operations 2541 b and 2543 b of the apps 2541and 2543. For example, the execution manager module 2553 may receiveinformation about the execution states of the operations 2541 b and 2543b from the apps 2541 and 2543. When the execution state of each of theoperations 2541 b and 2543 b is, for example, a partial landing state(e.g., when parameters utilized for the operations 2541 b and 2543 b arenot input), the execution manager module 2553 may transmit informationabout a suspended state to the intelligent agent 2551. The intelligentagent 2551 may use the received information to make a request for theuser to input relevant information (e.g., parameter information). By wayof another example, an utterance may be received from the user when theexecution state of each of the operations 2541 b and 2543 b is anoperable state, and the execution manager module 2553 may transmitinformation about the executed apps 2541 and 2543 and information aboutthe execution state of the apps 2541 and 2543 to the intelligent agent2551. The intelligent agent 2551 may receive parameter information ofthe user's utterance through the intelligent server 2600 and maytransmit the received parameter information to the execution managermodule 2553. The execution manager module 2553 may change the parametersof the operations 2541 b and 2543 b into new parameters using thereceived parameter information.

According to an embodiment, the execution manager module 2553 maytransmit the parameter information included in the path rule to the apps2541 and 2543. When the plurality of apps 2541 and 2543 is sequentiallyexecuted according to the path rule, the execution manager module 2553may transmit the parameter information included in the path rule fromone app to another app.

According to an embodiment, the execution manager module 2553 mayreceive a plurality of path rules. The execution manager module 2553 mayselect the plurality of path rules based on the user's utterance. Forexample, when the user's utterance specifies one app 2541 to execute oneoperation 2541 b but does not specify the other app 2543 to execute theremaining operation 2543 b, the execution manager module 2553 mayreceive a plurality of different path rules in which the same app 2541(e.g., a gallery app) to execute one operation 2541 b is executed andthe other app 2543 (e.g., a message app or a telegram app) to executethe remaining operation 2543 b is executed. The execution manager module2553 may execute the same operations 2541 b and 2543 b (e.g., the samesuccessive operations 2541 b and 2543 b) of the plurality of path rules.When the same operation has been executed, the execution manager module2553 may display, on the display 2520, the state screen for selectingthe different apps 2541 and 2543 respectively included in the pluralityof path rules.

According to an embodiment, the intelligent service module 2555 mayinclude a context module 2555 a, a persona module 2555 b, or asuggestion module 2555 c.

The context module 2555 a may collect current states of the apps 2541and 2543 from the apps 2541 and 2543. For example, the context module2555 a may receive the context information indicating the current statesof the apps 2541 and 2543 and may collect the current states of the apps2541 and 2543.

The persona module 2555 b may manage user's personal information usingthe user terminal 2500. For example, the persona module 2555 b maycollect usage information and performance results of the user terminal2500 to manage the user's personal information.

The suggestion module 2555 c may predict a user's intention and mayrecommend a command to the user. For example, the suggestion module 2555c may recommend the command to the user in consideration of the currentstate of the user (e.g., time, place, occasion, or app).

Referring to FIG. 27 , the user terminal 2500 may receive the user inputto execute an intelligent app (e.g., a voice recognition app)interworking with the intelligent agent 2551.

According to an embodiment, the user terminal 2500 may execute theintelligent app including voice recognition when activation of thehardware key 2512 is detected. For example, in response, the userterminal 2500 may display a user interface (UI) 2521 of the intelligentapp on the display 2520 when the user input is received via the hardwarekey 2512. Similarly, the user may touch a voice recognition button 2521a in the UI 2521 of the intelligent app in order to enable voice input2511 a and 2511 b when the UI 2521 of the intelligent app is displayedon the display 2520. By way of another example, the user maycontinuously press (e.g., long-press) the hardware key 2512 to activatethe voice input 2511 b function.

According to an embodiment, the user terminal 2500 may execute theintelligent app for recognizing the voice in response to inputs detectedvia microphones 2513 a and 2513 b. For example, the user terminal 2500may display the UI 2521 of the intelligent app on the display 2520 whenan input 2511 a is detected including a designated voice input command(e.g., wake up!), which is detected via the microphones 2513 a and 2513b.

FIG. 28 is a block diagram illustrating an intelligent server of anintegrated intelligent system 251 according to an embodiment of thepresent disclosure.

Referring to FIG. 28 , the intelligent server 2600 may include anautomatic speech recognition (ASR) module 2610, a natural languageunderstanding (NLU) module 2620, a path planner module 2630, a dialoguemanager (DM) module 2640, a natural language generator (NLG) module2650, or a text to speech (TTS) module 2660.

The NLU module 2620 or the path planner module 2630 of the intelligentserver 2600 may generate path rules.

According to an embodiment, the ASR module 2610 may convert the userinput received from the user terminal 2500 into text data.

According to an embodiment, the ASR module 2610 may convert the userinput received from the user terminal 2500 into text data. For example,the ASR module 2610 may include an utterance recognition module. Theutterance recognition module may include an acoustic model and alanguage model. For example, the acoustic model may include informationabout utterance, and the language model may include information aboutunit phoneme information and a combination of the unit phonemicinformation. The utterance recognition module may convert the user'sutterance into text data using the information about utterance and theunit phonemic information. The information about the acoustic model andthe language model may be stored in, for example, an automatic speechrecognition database (ASR DB) 2611.

According to an embodiment, the NLU module 2620 may perform syntacticanalysis or semantic analysis to ascertain a user's intention. Thesyntactic analysis may divide the user input into syntactic units (e.g.,words, phrases, morphemes, etc.) and may determine what syntacticelements the divided units have. The semantic analysis may be performedusing semantic matching, rule matching, formula matching, or the like.Accordingly, the NLU module 2620 may obtain a parameter (or slot) forthe user to express any one domain or the intention.

According to an embodiment, the NLU module 2620 may determine the user'sintention and the parameter using a matching rule divided into thedomain, the intention, and the parameter (or slot) for ascertaining theintention. For example, the one domain (e.g., alarm) may include aplurality of intentions (e.g., alarm setting, alarm cancellation, etc.),and one intention may include a plurality of parameters (e.g., time,number of repetitions, alarm tone, etc.). A plurality of rules mayinclude, for example, one or more essential element parameters. Thematching rule may be stored in the NLU DB 2621.

According to an embodiment, the NLU module 2620 may recognize themeaning of a word extracted from the user input using linguisticcharacteristics (e.g., syntactic elements) such as a morpheme, a phrase,etc., and may determine the user's intention by matching the recognizedmeaning of the word with the domain and the intention. For example, theNLU module 2620 may determine the user's intention by calculating howmany words extracted from the user input are included in each domain andintention. According to an embodiment, the NLU module 2620 may determinethe parameter of the user input using words used to understand theintention. According to an embodiment, the NLU module 2620 may determinethe user's intention using the NLU DB 2621 in which linguisticcharacteristics for ascertaining the intention of the user input arestored. According to another embodiment, the NLU module 2620 maydetermine the user's intention using a personal language model (PLM).For example, the NLU module 2620 may determine the user's intentionusing personalized information (e.g., a contact list or a music list).The PLM may be stored in, for example, the NLU DB 2621. According to anembodiment, not only the NLU module 2620 but also the ASR module 2610may recognize a user's voice by referring to the PLM stored in the NLUDB 2621.

According to an embodiment, the NLU module 2620 may generate the pathrule based on the intention and parameter of the user input. Forexample, the NLU module 2620 may select an app to be executed based onthe intention of the user input, and may determine an operation to beperformed in the selected app. The NLU module 2620 may generate the pathrule by determining the parameter corresponding to the determinedoperation. According to an embodiment, the path rule generated by theNLU module 2620 may include information about the app to be executed,the operation to be performed in the app, and the parameter utilized toperform the operation.

According to an embodiment, the NLU module 2620 may generate one pathrule or a plurality of path rules based on the intention and parameterof the user input. For example, the NLU module 2620 may receive a pathrule set corresponding to the user terminal 2500 from the path plannermodule 2630, and may map the intention and parameter of the user inputonto the received path rule set to determine the path rule.

According to another embodiment, the NLU module 2620 may generate onepath rule or a plurality of path rules by determining the app to beexecuted, the operation to be performed in the app, and the parameterutilized to execute the operation based on the intention and parameterof the user input. For example, the NLU module 2620 may generate thepath rule by arranging the app to be executed and the operation to beexecuted in the app in an ontology or graph model form according to theintention of the user input using the information of the user terminal2500. The generated path rule may be stored in a path rule database (PRDB) 2631 through, for example, the path planner module 2630. Thegenerated path rule may be added to the path rule set of the PR DB 2631.

According to an embodiment, the NLU module 2620 may select at least onepath rule among the generated plurality of path rules. For example, theNLU module 2620 may select an optimal path rule from the plurality ofpath rules. By way of another example, the NLU module 2620 may selectthe plurality of path rules when some operations are specified based onuser's utterance. The NLU module 2620 may determine one of the pluralityof path rules by a user's further input.

According to an embodiment, the NLU module 2620 may transmit the pathrule to the user terminal 2500 in response to the user input. Forexample, the NLU module 2620 may transmit one path rule corresponding tothe user input to the user terminal 2500. By way of another example, theNLU module 2620 may transmit the plurality of path rules correspondingto the user input to the user terminal 2500. The plurality of path rulesmay be generated by the NLU module 2620, for example, when someoperations are specified based on the user's utterance.

According to an embodiment, the path planner module 2630 may select atleast one path rule among the plurality of path rules.

According to an embodiment, the path planner module 2630 may transmitthe path rule set including the plurality of path rules to the NLUmodule 2620. The plurality of path rules of the path rule set may bestored in the form of a table in the PR DB 2631 connected to the pathplanner module 2630. For example, the path planner module 2630 maytransmit the path rule set corresponding to the information (e.g., OSinformation, app information, etc.) of the user terminal 2500 receivedfrom the intelligent agent 2551, to the NLU module 2620. The tablestored in the PR DB 2631 may be stored, for example, for each domain ordomain version.

According to an embodiment, the path planner module 2630 may select onepath rule or the plurality of path rules from the path rule set, and maytransmit the selected path rules to the NLU module 2620. For example,the path planner module 2630 may select one path rule or the pluralityof path rules by matching the user's intention and parameter with thepath rule set corresponding to the user terminal 101, and may transmitthe selected path rule to the NLU module 2620.

According to an embodiment, the path planner module 2630 may generateone path rule or the plurality of path rules using the user's intentionand parameter. For example, the path planner module 2630 may generateone path rule or a plurality of path rules by determining the app to beexecuted and the operation to be executed in the app based on the user'sintention and parameter. According to an embodiment, the path plannermodule 2630 may store the generated path rule in the PR DB 2631.

According to an embodiment, the path planner module 2630 may store thepath rule generated by the NLU module 2620 in the PR DB 2631. Thegenerated path rule may be added to the path rule set stored in the PRDB 2631.

According to an embodiment, the table stored in the PR DB 2631 mayinclude the plurality of path rules or the plurality of path rule sets.The plurality of path rules or the plurality of path rule sets mayreflect the kind, version, type, or characteristics of the deviceperforming each path rule.

According to an embodiment, the DM module 2640 may determine whether theuser's intention ascertained by the NLU module 2620 is clear. Forexample, the DM module 2640 may determine whether the user's intentionis clear based on whether information of the parameter is sufficient.The DM module 2640 may determine whether the parameter ascertained bythe NLU module 2620 is sufficient to perform a task. According to anembodiment, the DM module 2640 may perform feedback that requestsinformation from the user when the user's intention is not clear. Forexample, the DM module 2640 may perform feedback that makes a requestfor information about the parameter for ascertaining the user'sintention.

According to an embodiment, the DM module 2640 may include a contentprovider module. When an operation can be performed based on theintention and parameter ascertained by the NLU module 2620, the contentprovider module may generate a result of performing the taskcorresponding to the user input. According to an embodiment, the DMmodule 2640 may transmit the result generated by the content providermodule to the user terminal 2500 in response to the user input.

According to an embodiment, the NLG module 2650 may change designatedinformation into a text form. The information changed into the text formmay be provided in the form of a natural language utterance. Thedesignated information changed into the text form may be, for example,information about additional inputs, information used to guide thecompletion of the operation corresponding to the user input, orinformation (e.g., feedback information about the user input) used toguide the additional inputs of the user. The information changed intothe text form may be transmitted to the user terminal 2500 to bedisplayed on the display 2520, or may be transmitted to the TTS module2660 to be changed into a voice form.

According to an embodiment, the TTS module 2660 may change theinformation in the text form into the information in the voice form. TheTTL module 2660 may receive the information in the text form from theNLG module 2650, may convert the information in the text form into theinformation in the voice form, and may transmit resultant information tothe user terminal 2500. The user terminal 2500 may output theinformation in the voice form to the speaker 2530.

According to an embodiment, the NLU module 2620, the path planner module2630, and the DM module 2640 may be implemented as a single module. Forexample, the NLU module 2620, the path planner module 2630, and the DMmodule 2640 may be implemented as the single module to determine theuser's intention and the parameter, and may generate a response (e.g.,the path rule) corresponding to the determined user's intention and theparameter. Accordingly, the generated response may be transmitted to theuser terminal 2500.

FIG. 29 is a diagram illustrating a method of generating a path rule ofa path planner module according to an embodiment of the presentdisclosure.

Referring to FIG. 29 , the NLU module 2620 according to an embodimentmay divide the function of an app into unit operations (A to F), and maystore them in the PR DB 2631. For example, the NLU module 2620 may storea path rule set including a plurality of path rules A-B1-C1, A-B1-C2,A-B1-C3-D-F, and A-B1-C3-D-E-F divided into the unit operations in thePR DB 2631.

According to an embodiment, the PR DB 2631 of the path planner module2630 may store the path rule set for performing the function of the app.The path rule set may include a plurality of path rules including aplurality of operations. The plurality of path rules may be sequentiallyarranged according to a parameter inputted to each of the plurality ofoperations. According to an embodiment, the plurality of path rules maybe configured in an ontology or graph model form to be stored in the PRDB 2631.

According to an embodiment, the NLU module 2620 may select the optimalpath rules A-B1-C3-D-F among the plurality of path rules A-B1-C1,A-B1-C2, A-B1-C3-D-F, and A-B1-C3-D-E-F corresponding to the intentionand parameter of the user input.

According to an embodiment, the NLU module 2620 may transmit a pluralityof rules to the user terminal 2500 when there is no path rule completelymatched with the user input. For example, the NLU module 2620 may selecta path rule (e.g., A-B1) that is partially corresponds to the userinput. The NLU module 2620 may select one or more path rules (e.g.,A-B1-C1, A-B1-C2, A-B1-C3-D-F, and A-B1-C3-D-E-F) that include the pathrule (e.g., A-B1) partially corresponding to the user input, and maytransmit the selected path rule to the user terminal 2500.

According to an embodiment, the NLU module 2620 may select one of theplurality of path rules based on the additional input to the userterminal 2500, and may transmit the selected one path rule to the userterminal 2500. For example, the NLU module 2620 may select the one pathrule (e.g., A-B1-C3-D-F) among the plurality of path rules (e.g.,A-B1-C1, A-B1-C2, A-B1-C3-D-F, and A-B1-C3-D-E-F) according to the userinput (e.g., an input to select C3) additionally input to the userterminal 2500, and may transmit the selected path rule to the userterminal 2500.

According to still another embodiment, the NLU module 2620 may determinethe user's intention or the parameter corresponding to the user input(e.g., the input to select C3) additionally input to the user terminal2500 via the NLU module 2620, and may transmit the determined user'sintention or parameter to the user terminal 2500. The user terminal 2500may select one path rule (e.g., A-B1-C3-D-F) among the plurality of pathrules (e.g., A-B1-C1, A-B1-C2, A-B1-C3-D-F, and A-B1-C3-D-E-F) based onthe transmitted intention or parameter.

Accordingly, the user terminal 2500 may complete the operations of theapps 2541 and 2543 by the selected one path rule.

According to an embodiment, when the intelligent server 2600 receivesthe user input that lacks information, the NLU module 2620 may generatea path rule partially corresponding to the received user input. Forexample, the NLU module 2620 may transmit {circle around (1)} thepartially corresponding path rule to the intelligent agent 2551. Theintelligent agent 2551 may transmit {circle around (2)} the partiallycorresponding path rule to the execution manager module 2553, and theexecution manager module 2553 may execute a first app 2541 according tothe path rule. The execution manager module 2553 may transmit {circlearound (3)} information about insufficient parameter to the intelligentagent 2551 while executing the first app 2541. The intelligent agent2551 may request an additional input from the user using the informationabout an insufficient parameter. When the additional input is received{circle around (4)} by the user, the intelligent agent 2551 may transmitand process the received additional input to the intelligent server2600. The NLU module 2620 may generate the path rule added based oninformation about the intention and parameter of the additionally inputuser input, and may transmit {circle around (5)} the generated path ruleto the intelligent agent 2551. The intelligent agent 2551 may transmit{circle around (6)} the path rule to the execution manager module 2553{circle around (6)} to execute a second app 2543.

According to an embodiment, when the user input on which someinformation is missing is received at the intelligent server 2600, theNLU module 2620 may transmit a user information request to thepersonalization information server 2700. The personalization informationserver 2700 may transmit information of a user who inputs the user inputstored in the persona database, to the NLU module 2620. The NLU module2620 may use the user information to select the path rule correspondingto the user input in which some operations are missing. Accordingly,even though the user input on which some information is missing isreceived at the intelligent server 2600, the NLU module 2620 may requestthe missing information to receive a further input, or may use the userinformation to determine the path rule corresponding to the user input.

FIG. 30 is a diagram illustrating a case in which a context module of aprocessor according to an embodiment of the present disclosure collectscurrent states.

Referring to FIG. 30 , when receiving □ a context request from theintelligent agent 2551, the context module 2555 a may request □ contextinformation indicating the current states of the apps 2541 and 2543 fromthe apps 2541 and 2543. According to an embodiment, the context module2555 a may receive □ the context information from the apps 2541 and 2543and may transmit □ the received context information to the intelligentagent 2551.

According to an embodiment, the context module 2555 a may receive aplurality of pieces of context information through the apps 2541 and2543. For example, the context information may be information about themost recently executed apps 2541 and 2543. By way of another example,the context information may be information about the current states ofthe apps 2541, 2543 (e.g., information about a photo when the photo isviewed in the gallery).

According to an embodiment, the context module 2555 a may receivecontext information indicating the current state of the user terminal2500 from the device platform as well as the apps 2541 and 2543. Thecontext information may include general context information, usercontext information, or device context information.

The general context information may include general context informationof the user terminal 2500. The general context information may beconfirmed through an internal algorithm by receiving data through asensor hub of the device platform. For example, the general contextinformation may include information about a current time and space. Theinformation about the current time and space may include, for example,information about a current time and a current location of the userterminal 2500. The current time may be confirmed through time on theuser terminal 2500, and the information about the current location maybe confirmed through a GPS. By way of another example, the generalcontext information may include information about physical movement. Theinformation about physical movement may include, for example,information about walking, running, driving, and the like. Theinformation about physical movement may be confirmed through a motionsensor. The information about driving may be confirmed through themotion sensor, and boarding and parking may be confirmed by sensing aBluetooth connection in a vehicle. By way of still another example, thegeneral context information may include user activity information. Theuser activity information may include, for example, information aboutcommuting, shopping, traveling, and the like. The user activityinformation may be confirmed using information about a user or a placewhich is registered in a database by an app.

The user context information may include information about the user. Forexample, the user context information may include information aboutemotional states of the user. The information about the emotional statesmay include, for example, information about user's happiness, sadness,anger, and the like. By way of another example, the user contextinformation may include information about the current state of the user.The information about the current state may include information about,for example, interest, intention, etc. (e.g., shopping).

The device context information may include information about the stateof the user terminal 2500. For example, the device context informationmay include information about path rules executed by the executionmanager module 2553. By way of another example, the device informationmay include information about a battery. The information on the batterymay be confirmed through, for example, the state of charge and dischargeof the battery. By way of still another example, the device informationmay include information about a connected device and network. Theinformation about the connected device may be confirmed, for example,via a communication interface to which the device is connected.

FIG. 31 is a diagram illustrating a case in which a persona moduleaccording to an embodiment of the present disclosure manages userinformation.

Referring to FIG. 31 , the persona module 2555 b may receive informationof the user terminal 2500 from the apps 2541 and 2543, the executionmanager module 2553, or the context module 2555 a. The apps 2541 and2543 and the execution manager module 2553 may store information about aresult obtained by executing operations of the apps 2541 b and 2543 b,in an operation log database. The context module 2555 a may store theinformation about the current state of the user terminal 2500 in acontext database. The persona module 2555 b may receive the storedinformation from the operation log database or the context database. Thedata stored in the operation log database and the context database maybe analyzed by, for example, an analysis engine and may be transmittedto the persona module 2555 b.

According to an embodiment, the persona module 2555 b may transmit theinformation received from the apps 2541 and 2543, the execution managermodule 2553, or the context module 2555 a, to the suggestion module 2555c. For example, the persona module 2555 b may transmit the data storedin the operation log database or the context database to the suggestionmodule 2555 c.

According to an embodiment, the persona module 2555 b may transmit theinformation received from the apps 2541 and 2543, the execution managermodule 2553, or the context module 2555 a, to the personalizationinformation server 2700. For example, the persona module 2555 b mayperiodically transmit the data accumulated and stored in the operationlog database or the context database to the personalization informationserver 2700.

According to an embodiment, the persona module 2555 b may transmit thedata stored in the operation log database or the context database to thesuggestion module 2555 c. The user information generated by the personamodule 2555 b may be stored in a persona database. The persona module2555 b may periodically transmit the user information stored in thepersona database to the personalization information server 2700.According to an embodiment, the information transmitted by the personamodule 2555 b to the personalization information server 2700 may bestored in the persona database. The personalization information server2700 may infer user information utilized for generating the path rule ofthe intelligent server 2600 using the information stored in the personadatabase.

According to an embodiment, the user information inferred using theinformation transmitted by the persona module 2555 b may include profileinformation or preference information. The profile information or thepreference information may be inferred through a user's account andaccumulated information.

The profile information may include personal information of a user. Forexample, the profile information may include demographic information ofthe user. The demographic information may include, for example, theuser's gender, age, and the like. By way of another example, the profileinformation may include life event information. The life eventinformation may be inferred by comparing log information with a lifeevent model, and may be reinforced by analyzing a behavior pattern. Byway of another example, the profile information may include interestinformation. The interest information may include, for example, ashopping item of interest, an area of interest (e.g., sports, politics,etc.). By way of another example, the profile information may includeactivity area information. The activity area information may includeinformation about, for example, home, work place, and the like. Theactive area information may include information about a location wherethe priority is recorded based on a cumulative residence time and avisit frequency, as well as information about a location of the place.By way of another example, the profile information may include activitytime information. The activity time information may include informationabout, for example, a wake-up time, a commute time, a sleeping time, andthe like. The information about the commute time may be inferred usingthe activity area information (e.g., information about home and workplace). The information about the sleeping time may be inferred throughan unused time of the user terminal 2500.

The preference information may include user preference information. Forexample, the preference information may include information about apppreference. The app preference may be inferred, for example, through ausage history of the app (e.g., a usage history for each time andplace). The app preference may be used to determine an app to beexecuted according to a user's current state (e.g., time and place). Inanother example, the preference information may include informationabout contact preference. The contact preference may be inferred byanalyzing a contact frequency of the contact (e.g., a frequency ofcontact for each time or place). The contact preference may be used todetermine the contact to contact according to the current state of theuser (e.g., contact for duplicate names). In another example, thepreference information may include setting information. The settinginformation may be inferred by analyzing information about a settingfrequency of a specific setting value (e.g., a frequency of setting as asetting value for each time or place). The setting information may beused to set a specific setting value according to the user's currentstate (e.g., time, place, or occasion). In another example, thepreference information may include location preference. The placepreference may be inferred, for example, through a visit history (e.g.,a visit history for each time) of a specific place. The locationpreference may be used to determine a visited location according to theuser's current state (e.g., time). In another example, the preferenceinformation may include command preference. The command preference maybe inferred through, for example, a command usage frequency (e.g., afrequency of use for each time and place). The command preference may beused to determine a command pattern to be used according to the user'scurrent state (e.g., time and location). In particular, the commandpreference may include information about menus most frequently selectedby the user in the current state of the app that is executed byanalyzing the log information.

FIG. 32 is a block diagram illustrating a suggestion module according toan embodiment of the present disclosure.

Referring to FIG. 32 , the suggesting module 2555 c may include a hintproviding module 2555 c_1, a context hint generating module 2555 c_2, acondition checking module 2555 c_3, a condition model module 2555 c_4, areuse hint generating module 2555 c_5, or an introduction hintgenerating module 2555 c_6.

According to an embodiment, the hint providing module 2555 c_1 mayprovide a hint to the user. For example, the hint providing module 2555c_1 may receive a hint generated from the context hint generating module2555 c_2, the reuse hint generating module 2555 c_5, or the introductionhint generating module 2555 c_6 to provide the generated hint to theuser.

According to an embodiment, the context hint generation module 2555 c_2may generate a hint that can be recommended according to the currentstate through the condition checking module 2555 c_3 or the conditionmodel module 2555 c_4. The condition checking module 2555 c_3 mayreceive information corresponding to the current state via theintelligent service module 2555, and the condition model module 2555 c_4may set a condition model using the received information. For example,the condition model module 2555 c_4 may determine the time, location,occasion, and application in use at the time of providing the hint tothe user, and may provide hints that are highly likely to be used underthe corresponding condition to the user in descending order of priority.

According to an embodiment, the reuse hint generation module 2555 c_5may generate a hint that can be recommended in consideration of thefrequency of use according to the current state. For example, the reusehint generation module 2555 c_5 may generate a hint in consideration ofthe usage pattern of the user.

According to an embodiment, the introduction hint generation module 2555c_6 may generate a hint to introduce a new function or a functionfrequently used by another user to the user. For example, the hint tointroduce the new function may include an introduction to theintelligent agent 2551 (e.g., an operation method).

According to another embodiment, the context hint generation module 2555c_2, the condition checking module 2555 c_3, the condition model module2555 c_4, the reuse hint generation module 2555 c_5, or the introductionhint generation module 2555 c_6 of the suggestion module 2555 c may beincluded in the personalization information server 2700. For example,the hint providing module 2555 c_1 of the suggesting module 2555 c mayreceive the hint from the context hint generating module 2555 c_2, thereuse hint generating module 2555 c_5, or the introduction hintgenerating module 2555 c_6 of the user personalization informationserver 2700, and may provide the received hint to the user.

According to an embodiment, the user terminal 2500 may provide the hintaccording to the following sequence of processes. For example, whenreceiving □ a hint providing request from the intelligent agent 2551,the hint providing module 2555 c_1 may transmit □ the hint generatingrequest to the context hint generating module 2555 c_2. When receivingthe hint generation request, the context hint generation module 2555 c_2may use □ the condition checking module 2555 c_3 to receive □information corresponding to the current state from the context module2555 a and the persona module 2555 b. The condition checking module 2555c_3 may transmit {circle around (5)} the received information to thecondition model module 2555 c_4, and the condition model module 2555 c_4may assign priorities to the hints in descending order of usabilityunder the condition among the hints provided to the user using theinformation. The context hint generation module 2555 c_2 may confirm{circle around (6)} the condition and may generate the hintcorresponding to the current state. The context hint generation module2555 c_2 may transmit {circle around (7)} the generated hint to the hintproviding module 2555 c_1. The hint providing module 2555 c_1 may arraythe hints according to a designated rule and may transmit {circle around(8)} the hint to the intelligent agent 2551.

According to an embodiment, the hint providing module 2555 c_1 maygenerate a plurality of context hints and may assign priorities to theplurality of context hints according to the designated rule. Accordingto an embodiment, the hint providing module 2555 c_1 may first provide,to the user, the context hint with a higher priority among the pluralityof context hints.

According to an embodiment, the user terminal 2500 may suggest a hintaccording to the frequency of use. For example, when receiving {circlearound (1)} the hint providing request from the intelligent agent 2551,the hint providing module 2555 c_1 may transmit {circle around (2)} thehint generation request to the reuse hint generating module 2555 c_5.When receiving the hint generation request, the reuse hint creationmodule 2555 c_5 may receive {circle around (3)} the user informationfrom the persona module 2555 b. For example, the reuse hint generationmodule 2555 c_5 may receive the path rule included in the user'spreference information of the persona module 2555 b, the parameterincluded in the path rule, the execution frequency of the app, and timeand space information in which the app is used. The reuse hintgeneration module 2555 c_5 may generate the hint corresponding to thereceived user information. The reuse hint generation module 2555 c_5 maytransmit {circle around (4)} the generated hint to the hint providingmodule 2555 c_1. The hint providing module 2555 c_1 may array the hintsand may transmit {circle around (5)} the hints to the intelligent agent2551.

According to an embodiment, the user terminal 2500 may suggest a hintfor a new function. For example, when receiving {circle around (1)} ahint providing request from the intelligent agent 2551, the hintproviding module 2555 c_1 may transmit {circle around (2)} the hintproviding request to the introduction hint generating module 2555 c_6.The introduction hint generation module 2555 c_6 may transmit {circlearound (3)} an introduction hint providing request to the suggestionserver 2800 to receive {circle around (4)} information about a functionto be introduced from the suggestion server 2800. For example, thesuggestion server 2800 may store the information about the function tobe introduced, and a hint list for the function to be introduced may beupdated by a service operator. The introduction hint generation module2555 c_6 may transmit {circle around (5)} the generated hint to the hintproviding module 2555 c_1. The hint providing module 2555 c_1 may arraythe hints and may transmit {circle around (6)} the arrayed hints to theintelligent agent 2551.

Accordingly, the suggestion module 2555 c may provide, to the user, thehint generated in the context hint generation module 2555 c_2, the reusehint generation module 2555 c_5, or the introduction hint generationmodule 2555 c_6. For example, the suggestion module 2555 c may displaythe generated hint in an app that operates the intelligent agent 2551,and may receive an input to select the hint from the user through theapp.

FIG. 33 is a diagram illustrating an example in which an electronicdevice according to an embodiment of the present disclosure provides ahint.

Referring to FIG. 33 , when receiving an incoming call, the electronicdevice may provide a hint corresponding to the reception of the incomingcall according to the above-described embodiment. For example, the hintmay be displayed on a screen of the electronic device as a pop-up window3301 including information to guide a user input such as saying “Answerthe phone on Hi Bixby speaker phone” or saying “Answer the phone” on thespeaker phone while pressing button. For example, when the user says“Answer the phone on Hi Bixby speaker phone” or says “Answer the phone”while touching a button 3303, a speakerphone function of the electronicdevice may be executed.

FIG. 34 is a diagram illustrating a system for executing the function ofan electronic device (e.g., the user terminal 2500) according to variousembodiments.

Referring to FIG. 34 , when receiving an utterance input or text inputof a user, the electronic device 3401 may transmit a corresponding voiceor text to a server 3411 (e.g., the intelligent server 2600) via anintelligent agent 3405 (e.g., the intelligent agent 2551).

According to an embodiment, when receiving the voice from the electronicdevice 3401, an ASR module 3413 (e.g., the ASR module 2610) of theserver 3411 may convert the received voice into text data, and an NLUmodule 3415 (e.g., the NLU module 2620) may determine how to process theconverted text data. For example, the NLU module 3415 may ascertain auser's intention by performing syntactic analysis or semantic analysisaccording to the content of the text data. For example, the NLU module3415 may determine the user's intention and the parameter using amatching rule divided into the domain, intention, and the parameter (orslot) utilized for ascertaining the intention. A response generator 3419may generate a response corresponding to the voice received from theelectronic device 3401 based on the analysis result of the NLU module3415, and may transmit the generated response to the intelligent agent3405 of the electronic device 3401. For example, a content providingmodule 3425 of the response generator 3419 may generate a resultobtained by performing a task corresponding to the user input based onthe intention and the parameter ascertained by the NLU module 3415. Forexample, a path planner module 3427 (e.g., the path planner module 2630)of the response generator 3419 may generate the path rule correspondingto the user input based on the intention and the parameter ascertainedby the NLU module 3415.

According to an embodiment, when receiving the text data from theelectronic device 3401, the NLU module 3415 of the server 3411 maydetermine how to process the text data. For example, the NLU module 3415may determine the user's intent and the parameter using the matchingrule divided into the domain, the intention, and the parameter (or slot)utilized for ascertaining the intention. The response generator 3419 maygenerate a response corresponding to the text data received from theelectronic device 3401 based on the analysis result of the NLU module3415, and may transmit the generated response to the intelligent agent3405 of the electronic device 3401. For example, the content providingmodule 3425 of the response generator 3419 may generate the resultobtained by performing the task corresponding to the user input based onthe intention and the parameter ascertained by the NLU module 3415. Forexample, the path planner module 3427 of the response generator 3419 maygenerate the path rule corresponding to the user input based on theintention and the parameter ascertained by the NLU module 3415.

The intelligent agent 3405 may transmit the response received from theresponse generator 3419 to an execution manager module 3407 (e.g., theexecution manager module 2553) to perform a function (also referred toas task) corresponding to an utterance based on the received response.For example, the execution manager module 3407 may sequentially performoperations corresponding to the utterance in an application 3409(referred to as app, e.g., the first app 2541 or the second app 2543)based on the received response. When the operations are completelyperformed, the intelligent agent 3405 may transmit and generate anoperation completion message to an NLG module 3421 (e.g., the NLG module2650). The NLG module 3421 may generate feedback suitable for anoccasion and may transmit the generated feedback to a TTS module 3423(e.g., the TTS module 2660). Next, the TTS module 3423 may transmitvoice data for the received feedback to the intelligent agent 3405. Theelectronic device 3401 may provide the voice data for the feedbackreceived from the TTS module 3423, to the user in the form of voiceand/or text.

According to an embodiment, the electronic device 3401 may include aclient content provider 3403 (hereinafter referred to as a client CP).The client CP 3403 may exist as a separate module in the electronicdevice 3401, and may be included in the intelligent agent 3405. Theclient CP 3403 may include a database for storing data associated withthe macro generated by the user, and may be synchronized in real timewith a personal language model (PLM) module 3417 in the server 3411.

For example, the macro may be a command shortcut that is mapped to arepresentative dialog for executing any one function of the electronicdevice to execute the function.

For example, the database included in the client CP 3403 may store atleast one command shortcut and at least one representative dialog mappedto the command shortcut. For example, the database may store the atleast one command shortcut and the at least one representative dialogmapped to the command shortcut, in the form of a table. For example, thecommand shortcut may be a command preset by a user, and may be stored astext data. For example, the representative dialog may correspond to thepath rule including information of the sequence of the states of theelectronic device for performing any one function (referred to as task)of the electronic device. The representative dialog may be a naturallanguage for executing any one function of the electronic device, andmay be stored as text data. The representative dialog may correspond tothe hint described above, and may be text data indicating a voice to beinput by the user so that the operation of the electronic device can beexecuted through a voice input. In addition, the representative dialogmay include terms related to the corresponding function, and theparameters included in the representative dialog may be changed.

For example, the PLM module 3417 (e.g., the ASR DB 2611) may storepersonalized utterance information of the user and/or the electronicdevice 3401, and the PLM module 3417 and the command shortcut may besynchronized in real time with the client CP 3403 at the time ofregistration, modification, and deletion of the macro of the commandshortcut by the user. For example, the command shortcut synchronized inreal time with the client CP 3403 of the PLM module 3417 may support toimprove the performance when the ASR module 3413 is operated.

FIG. 35 is a flowchart illustrating the operation of an electronicdevice (e.g., the user terminal 2500) for executing the function of theelectronic device according to various embodiments. Referring to FIG. 35, the electronic device may control the function execution of theelectronic device based on the macro generated by the user.

In operation 3510, the electronic device (e.g., the processor 2550 ofFIG. 26 ) may receive a user input.

According to an embodiment, the user input may be a voice input and/or atext input using a microphone, a touch on a touch screen, a usermanipulation on buttons (or keys), and/or a remote controller. Forexample, the user input may be a command shortcut that is set as a macroso as to execute any one function of the electronic device, or arepresentative dialog that is set so as to execute any one function ofthe electronic device. The function may include at least one unitoperation.

According to an embodiment, the electronic device may provide a hintthrough the touch screen and/or the speaker so that the user canprecisely select the function to be performed by the electronic device.For example, the hint may be that the representative dialog is providedin the form of text and/or voice to execute a specific function of theelectronic device. For example, in order to execute the specificfunction of the electronic device, the user may perform a user input,such as uttering the hint that is the representative utterance, touchingthe hint, inputting text corresponding to the hint, etc., based on thehint.

For example, the hint may be prepared to predict the user's intentionand perform the user command more smoothly. The hint can be used torecommend a user-customized command in consideration of the currentstate of the user among all the commands that the user can perform. Thecurrent state of the user may include time, place, and/or occasioninformation (TPO), and may include the type of an application currentlyexecuted in the electronic device.

For example, the hint may be an introduce hint with a purpose tointroduce a new function, a context information hint considering user'sTPO, or a reuse hint for suggesting a function frequently used by theuser.

In operation of the electronic device for generating the hint has beendescribed in detail in the above-described embodiments and thus furtherdescription thereof will be omitted.

In operation 3520, the electronic device (e.g., the processor 2550 ofFIG. 26 ) may confirm the representative dialog based on the textcorresponding to the user input.

According to an embodiment, when the user input is the voice input, theelectronic device may transmit the voice input to the server, and theserver that has received the voice input may convert the voice inputinto text data using an automatic speech recognition (ARS) technique andthen transmit the converted text data to the electronic device. Theelectronic device may select the representative dialog that is mapped tothe converted text data in the database.

According to an embodiment, when the user input is the text input, theelectronic device may select the representative dialog that is mapped tothe text data input in the database.

According to an embodiment, the database may be in the electronicdevice, and the database may store at least one command shortcut and atleast one representative dialog which is mapped to the command shortcut.For example, the database may be included in a client CP (e.g., clientCP 3403). The database may store the at least one command shortcut andthe at least one representative dialog that is mapped to the commandshortcut, in the form of a table. The command shortcut may be a commandpreset by a user, and may be stored as text data. The representativedialog may be a natural language for executing a specific function ofthe electronic device, and may be stored as text data. In addition, therepresentative dialog may include terms related to the function.

In operation 3530, the electronic device (e.g., the processor 2550 ofFIG. 26 ) may execute operations of the function corresponding to theuser input using the path rule determined based on the representativedialog.

According to an embodiment, the electronic device may transmit therepresentative dialog to a server (e.g., the intelligent server 2600).The server may determine the path rule for executing the functioncorresponding to the representative dialog using the representativedialog, and may transmit the determined path rule to the electronicdevice. For example, the path rule may be that the electronic devicedivides the function of the application into unit operations. Forexample, the server may confirm a preset path rule corresponding to therepresentative dialog. By way of another example, the server maygenerate the path rule corresponding to the representative dialog,including an NLU module (e.g., the NLU module 2620) and a path plannermodule (e.g., the path planner module 2630). Since the detailedoperation for generating (extracting) the path rule has been describedin the above embodiments, detailed description thereof is omitted.

The electronic device may execute the operations corresponding to thepath rule based on the path rule.

FIG. 36 is a flowchart illustrating a macro setting operation forexecuting the function of an electronic device (e.g., the user terminal2500) according to various embodiments.

According to an embodiment, a user of the electronic device may utterthe entire dialog corresponding to the hint so that the electronicdevice performs the function corresponding to the hint. However,referring to FIG. 36 , by performing an utterance input of some words orsentences designated to correspond to the dialog or performing a touchinput, the electronic device may execute the function corresponding tothe hint. For example, the electronic device may provide a macrofunction capable of changing the hint (hereinafter, also referred to asa representative dialog) corresponding to the path rule for executingany one function of the electronic device into a short sentencedesignated by the user. For example, the electronic device may set somewords or sentences as the macro that executes any one function of theelectronic device. For example, the electronic device may set some wordsor sentences to be matched to the representative dialog set to executeany one function of the electronic device.

In operation 3610, the electronic device (e.g., the processor 2550 ofFIG. 26 ) may receive the user input for macro registration,modification, or deletion.

According to an embodiment, the user input may be received in one of adiverse number of formats, such as a voice input using a microphone, atouch input on a touch screen, a user manipulation on buttons (or keys),and/or an input using a remote controller.

In operation 3630, the electronic device (e.g., the processor 2550 ofFIG. 26 ) may store the macro in the database, modify the macro storedin the database, or delete the macro stored in the database, based onthe user input.

According to an embodiment, when receiving the user input for theregistration of the macro, the electronic device may store a commandshortcut desired by the user and a representative dialog set toperforming the corresponding function in the database. For example, theelectronic device may extract the representative dialog associated withthe function performed by the electronic device, and may match therepresentative dialog to the command shortcut to store the mappedinformation in the database together with the command shortcut. Thecommand shortcut may be set according to the user input, and may bestored in the form of voice data or text data. The representative dialogmay be the content of the hint in the above-described embodiments or thecontent of the hint mapped with the path rule including information ofthe sequence of the states of the electronic device.

According to an embodiment, the electronic device may enable the commandshortcut to be modified or deleted. For example, the electronic devicemay receive the user input, and may modify the command shortcut tomodify the macro or delete the command shortcut to delete the macro.

According to an embodiment, the electronic device may generate a commandshortcut for a function that the user frequently uses, and may store thegenerated command shortcut in the database.

According to an embodiment, the database may be included in theelectronic device.

In operation 3650, the electronic device (e.g., the processor 2550 ofFIG. 26 ) may synchronize text data (i.e., text data corresponding tothe macro resulting from registration, modification, or deletion of themacro) with the server (e.g., the intelligent server 2600 or the ASR DB2611 of the intelligent server) in real time.

According to an embodiment, the electronic device may cause the commandshortcut stored in the database of the electronic device to besynchronized in real time with the server that communicates with theelectronic device, so that the recognition performance of the server maybe improved when the user utters the corresponding command shortcut orperforms a touch input.

FIGS. 37A and 37B are flowcharts illustrating an operation of executingthe function of an electronic device (e.g., the user terminal 2500)according to various embodiments.

Referring to FIGS. 37A and 37B, the electronic device may store arepresentative dialog set to perform any one function of the electronicdevice and a command shortcut mapped with the representative dialog. Therepresentative dialog may correspond to a path rule includinginformation of a sequence of states of the electronic device. Whenreceiving the user input, the electronic device may confirm whether theuser input is a command shortcut set as a macro, and may perform afunction corresponding to the command shortcut through communicationwith the server.

In operation 3710, the electronic device 3701 (e.g., the processor 2550of FIG. 26 ) may receive the user input.

According to an embodiment, the user input may be a voice input and/or atext input using a microphone, a touch on a touch screen, a usermanipulation on buttons (or keys), and/or a remote controller.

In operation 3715, the electronic device 3701 (e.g., the processor 2550of FIG. 26 ) may determine whether the received user input correspondsto a pre-stored command shortcut. When it is determined that thereceived user input corresponds to the pre-stored command shortcut inoperation 3715, the electronic device 3701 may execute an operation3720, and otherwise, the electronic device 3701 may execute an operation3745.

According to an embodiment, the electronic device may confirm whetherthe user input is the command shortcut in which the user input is set asa macro. The command shortcut may be stored in the database of theelectronic device according to the user in advance. For example, thecommand shortcut may be mapped with the representative dialog requestingto perform any one function (task) of the electronic device, and may bestored in advance in the database together with the representativedialog. For example, the representative dialog may be mapped to the pathrule including the information of the sequence of the states of theelectronic device for performing a function. For example, when the userinput is the voice input, it may be confirmed whether the text datacorresponding to the voice input corresponds to the pre-stored commandshortcut. For example, when the user input is the text input, it may beconfirmed whether the text input corresponds to the pre-stored commandshortcut.

In operation 3720, the electronic device 3701 (e.g., the processor 2550of FIG. 26 ) may confirm the representative dialog corresponding to thecommand shortcut.

According to an embodiment, in a case in which “Music cue” is preset andstored as the command shortcut that is mapped with “Play real timepopular chart in music player” that is a representative dialog of theelectronic device that executes the real time popular chart in the musicplayer, when receiving “Music cue” via the text or voice input of theuser, the electronic device may determine that the received user inputcorresponds to the pre-stored command shortcut, and may confirm “Playreal time popular chart in music player” that is the representativedialog corresponding to “Music cue”.

In operation 3725, the electronic device 3701 (e.g., the processor 2550of FIG. 26 ) may transmit the representative dialog to the server 3703(e.g., the intelligent server 2600).

According to an embodiment, the electronic device 3701 may transmit therepresentative dialog to the server 3703 using a wireless communicationcircuit.

In operation 3730, the server 3703 (e.g., the intelligent server 2600)may extract the path rule corresponding to the representative dialog.

According to an embodiment, the server 3703 may confirm the pre-set pathrule corresponding to the representative dialog.

According to another embodiment, the server 3703 may include the NLUmodule (e.g., the NLU module 2620) and the path planner module (e.g.,the path planner module 2630) to generate the path rule corresponding tothe representative dialog. Since the detailed operation of generating(extracting) the path rule has been described in the above-describedembodiments, a detailed description thereof will be omitted.

In operation 3735, the server 3703 (e.g., the intelligent server 2600)may transmit the extracted path rule to the electronic device 3701.

According to an embodiment, the server 3703 may transmit the path ruleto the electronic device 3701 using the wireless communication circuit.

In operation 3740, the electronic device 3701 (e.g., the processor 2550of FIG. 26 ) may execute an operation corresponding to the path rule.

According to an embodiment, the electronic device 3701 may include anexecution manager module (e.g., the execution manager module 2553) toperform the operation corresponding to the path rule.

In operation 3745, the electronic device 3701 (e.g., the processor 2550of FIG. 26 ) may transmit the user input to the server 3703 (e.g., theintelligent server 2600).

According to an embodiment, the electronic device 3701 may transmit theuser input to the server 3703 using the wireless communication circuit.

According to an embodiment, in a case in which “Music cue” is preset andstored as the command shortcut that is mapped with “Play real timepopular chart in music player” that is the representative dialog of theelectronic device that executes the real time popular chart in the musicplayer, when receiving “Play real time popular chart in music player”via the text or voice input of the user, the electronic device maydetermine that the received user input does not correspond to thepre-stored command shortcut, and may transmit the text or voice input tothe server.

In operation 3750, the server 3703 (e.g., the intelligent server 2600)may extract the path rule using the user input.

According to an embodiment, the user input may correspond to therepresentative dialog, and the server 3703 may confirm the preset pathrule corresponding to the representative dialog.

According to another embodiment, the server 3703 may include the NLUmodule (e.g., the NLU module 2620) and the path planner module (e.g.,the path planner module 2630) to generate the path rule corresponding tothe user input. Since the detailed operation for generating (extracting)the path rule has been described in the above-described embodiments, adetailed description thereof will be omitted.

After operation 3750, the server 3703 (e.g., the intelligent server2600) may execute operation 3735 described above.

FIG. 38 is a flowchart illustrating an operation of executing thefunction of an electronic device (e.g., the user terminal 2500)according to various embodiments.

Referring to FIG. 38 , a server may store a representative dialog set toperform any one function of the electronic device and a command shortcutmapped to the representative dialog, and the representative dialog maycorrespond to a path rule including information of a sequence of statesof the electronic device. The server that has received the user inputfrom the electronic device may confirm whether the user input is acommand shortcut set as a macro, and may transmit the path rulecorresponding to the command shortcut to the electronic device.

In operation 3810, an electronic device 3801 (e.g., the processor 2550of FIG. 26 ) may receive the user input.

According to an embodiment, the user input may be a voice input and/or atext input using a microphone, a touch on a touch screen, a usermanipulation on buttons (or keys), and/or a remote controller.

In operation 3815, the electronic device 3701 (e.g., the processor 2550of FIG. 26 ) may transmit the user input to the server 3803 (e.g., theintelligent server 2600).

According to an embodiment, the electronic device 3801 may transmit theuser input to the server 3803 using a wireless communication circuit.

In operation 3820, the server 3803 (e.g., intelligent server 2600) maydetermine whether the user input corresponds to a pre-stored commandshortcut.

According to an embodiment, the server may confirm a database thatstores at least one command shortcut having a macro function and atleast one representative dialog mapped to the command shortcut todetermine whether the user input is the command shortcut stored forperforming the macro function.

When it is determined that the user input corresponds to the pre-storedcommand shortcut, the server 3803 (e.g., intelligent server 2600) mayexecute operation 3825, and otherwise, the server 3803 may executeoperation 3845.

In operation 3825, the server 3803 (e.g., the intelligent server 2600)may confirm the representative dialog corresponding to the commandshortcut.

According to an embodiment, in a case in which “Music cue” is a presetstored command shortcut that is mapped with a representative dialog suchas “Play real time popular chart in music player,” which can be used toexecute the real time popular chart in the music player, when the userinput is “Music cue”, the server 3803 may determine that the user inputcorresponds to the pre-stored command shortcut, and may confirm “Playreal time popular chart in music player” that is the representativedialog corresponding to “Music cue”.

In operation 3830, the server 3803 (e.g., the intelligent server 2600)may extract the path rule using the representative dialog.

According to an embodiment, the server 3803 may confirm a pre-set pathrule corresponding to the representative dialog.

According to another embodiment, the server 3803 may include an NLUmodule (e.g., the NLU module 2620) and a path planner module (e.g., thepath planner module 2630) to generate the path rule corresponding to therepresentative dialog. Since the detailed operation of generating(extracting) the path rule has been described in the above-describedembodiments, a detailed description thereof will be omitted.

In operation 3835, the server 3803 (e.g., the intelligent server 2600)may transmit the path rule to the electronic device 3801 (e.g., theprocessor 2550 of FIG. 26 ).

According to an embodiment, the server 3803 may transmit the path ruleto the electronic device 3801 using a wireless communication circuit.

In operation 3840, the electronic device 3801 (e.g., the processor 2550of FIG. 26 ) may perform an operation corresponding to the path rule.

According to an embodiment, the electronic device 3801 may include anexecution manager module (e.g., execution manager module 2553) toperform the operation corresponding to the path rule.

In operation 3845, the server 3803 (e.g., the intelligent server 2600)may extract the path rule using the user input.

According to an embodiment, the user input may correspond to therepresentative dialog, and the server 3803 may check the pre-set pathrule corresponding to the representative dialog.

According to another embodiment, the server 3803 may include the NLUmodule (e.g., the NLU module 2620) and the path planner module (e.g.,the path planner module 2630) to generate the path rule corresponding tothe user input. Since the detailed operation of generating (extracting)the path rule has been described in the above-described embodiments, adetailed description thereof will omitted.

After operation 3845, the server 3803 (e.g., the intelligent server2600) may execute operation 3835 described above.

FIG. 39 is a diagram illustrating a system for executing the function ofan electronic device according to various embodiments.

Referring to FIG. 39 , depending on the user input, an electronic device3901 (e.g., the user terminal 2500) may register, modify, or deletemacros to execute any one function of the electronic device in a clientCP 3903 using a simple command shortcut. For example, the client CP 3903may include a database, and the electronic device may register, modify,or delete the macros in the database. The client CP 3903 and a server3905 (e.g., the intelligent server 2600) may be synchronized in realtime, and when the macro is registered, modified, or deleted in theclient CP 3903, a command shortcut having a macro function may besynchronized with a PLM module 3909 of the server 3905. For example,when an ASR module 3907 (e.g., the ASR module 2610) of the server 3905converts received utterance contents into text data, the PLM module 3909may improve the operation performance of the ASR module 3907 byproviding the command shortcut. When the command shortcut designated asthe macro by the user is input via voice, the electronic device 3901 maycommunicate with the ASR module 3907 of the server 3905 in real time toconfirm the utterance contents performed by the user as text data. Forexample, the electronic device 3901 may communicate with the ASR module3907 of the server 3905 in real time, and the ASR module 3907 mayconvert the received utterance contents into text data in real time tothereby transmit the text data to the electronic device 3901. Theelectronic device 3901 may check the database included in the client CP3903 to confirm the command shortcut corresponding to the text data inthe database, and may confirm the representative dialog mapped to thecommand shortcut. The client CP 3903 may transmit the representativedialog to an NLU module 3911 (e.g., the NLU module 2620) of the server3905 via the electronic device 3901, and the NLU module 3911 maydetermine how to process the representative dialog. For example, the NLUmodule 3911 may perform syntactic analysis or semantic analysis toascertain a user's intention according to the content of the text data.For example, the NLU module 3911 may determine the user's intention andthe parameter using a matching rule divided into domain, intention, andparameter (or slot) utilized for ascertaining the intention. A pathplanner module 3913 (also referred to as PP module) (e.g., the pathplanner module 2630) of the server 3905 may generate the path rulecorresponding to the user input based on the intention and parameterascertained by the NLU module 3911.

According to the above-described embodiment of FIG. 39 , although theclient CP 3903 has been described as a separate configuration from theelectronic device 3901, according to another embodiment, the client CP3903 may be included in the electronic device 3901 and may perform theabove-described operation.

According to the above-described embodiment of FIG. 39 , it has beendescribed that the server 3905 includes the ASR module 3907 and the PLMmodule 3909. However, according to another embodiment, although notshown, the electronic device 3901 includes the ASR module 3907 and thePLM module 3909 so that the operations of the ASR module 3907 and thePLM module 3909 can be executed in the electronic device 3091. Accordingto still another embodiment, although not shown, the electronic device3901 may include all of the ASR module 3907, the PLM module 3909, theNLU module 3911, and the PP module 3913 so that the operations of theASR module 3907, the PLM module 3909, the NLU module 3911, and the PPmodule 3913 can be executed in the electronic device 3091.

FIG. 40 is a flowchart illustrating an operation of executing thefunction of an electronic device (e.g., the user terminal 2500)according to various embodiments.

Referring to FIG. 40 , the electronic device may store a representativedialog set to perform any one function of the electronic device and acommand shortcut mapped to the representative dialog, and therepresentative dialog may correspond to a path rule includinginformation of a sequence of states of the electronic device. Whenreceiving a voice input corresponding to the command shortcut from auser, the electronic device may execute a function corresponding to thecommand shortcut via communication with the server.

In operation 4005, an electronic device 4001 (e.g., the processor 2550of FIG. 26 ) may receive the voice input from the user.

According to an embodiment, the electronic device may receive the voiceinput from the user via a microphone.

In operation 4007, the electronic device 4001 (e.g., processor 2550 ofFIG. 26 ) may transmit the received voice input to a server 4003 (e.g.,the intelligent server 2600).

According to an embodiment, the electronic device 4001 may transmit thevoice input to the server 4003 via a wireless communication circuit.

In operation 4009, a server 4003 (e.g., intelligent server 2600) mayconfirm text data corresponding to the voice input.

According to an embodiment, the server 4003 may convert the voice inputinto the text data using an automatic speech recognition (ASR)technique. For example, the server 4003 may include an ASR module (e.g.,the ASR module 2610) to convert the voice input to text data.

In operation 4011, the server 4003 (e.g., the intelligent server 2600)may transmit the text data to the electronic device 4001 (e.g., theprocessor 2550 of FIG. 26 ).

According to an embodiment, the server 4003 may transmit the text datato the electronic device 4001 via a wireless communication circuit.

In operation 4013, the electronic device 4001 (e.g., the processor 2550of FIG. 26 ) may determine whether the voice input corresponds to apre-stored command shortcut based on the text data.

According to an embodiment, the electronic device 4001 may confirmwhether the text data is the command shortcut set as a macro. Thecommand shortcut may be stored in the database of the electronic devicein advance. For example, the command shortcut may be mapped with therepresentative dialog requesting to perform any one function (task) ofthe electronic device, and may be stored in advance in the databasetogether with the representative dialog. For example, the representativedialog may be mapped to the path rule including the information of thesequence of the states of the electronic device for performing afunction.

According to an embodiment, when the voice input corresponds to thepre-stored command shortcut, the electronic device 4001 (e.g., theprocessor 2550 of FIG. 26 ) may confirm the representative dialogcorresponding to the command shortcut in operation 4015.

In operation 4017, the electronic device 4001 (e.g., the processor 2550of FIG. 26 ) may transmit the confirmed representative dialog to theserver 4003 (e.g., the intelligent server 2600).

According to an embodiment, the electronic device 4001 may transmit theconfirmed representative dialog to the server 4003 via a wirelesscommunication circuit.

In operation 4019, the server 4003 (e.g., the intelligent server 2600)may extract the path rule using the representative dialog.

According to an embodiment, the server 4003 may confirm a preset pathrule corresponding to the representative dialog.

According to another embodiment, the server 4003 may include an NLUmodule (e.g., the NLU module 2620) and a path planner module (e.g., thepath planner module 2630) to generate the path rule corresponding to therepresentative dialog. Since the detailed operation of generating(extracting) the path rule has been described in the above-describedembodiments, a detailed description thereof will be omitted.

In operation 4021, the server 4003 (e.g., the intelligent server 2600)may transmit the path rule to the electronic device 4001 (e.g., theprocessor 2550 of FIG. 26 ).

According to an embodiment, the server 4003 may transmit the path ruleto the electronic device 4001 via the wireless communication circuit.

In operation 4023, the electronic device 4001 (e.g., the processor 2550in FIG. 26 ) may perform an operation corresponding to the path rule.

According to an embodiment, the electronic device 4001 may include anexecution manager module (e.g., the execution manager module 2553) toperform the operation corresponding to the path rule.

FIG. 41 is a flowchart illustrating an operation of executing thefunction of an electronic device (e.g., the user terminal 2500)according to various embodiments. Referring to FIG. 41 , a server maystore a representative dialog set to perform any one function of theelectronic device and a command shortcut mapped to the representativedialog, and the representative dialog may correspond to a path ruleincluding information of a sequence of states of the electronic device.When receiving a voice input corresponding to the command shortcut fromthe electronic device, the server may extract a path rule correspondingto the command shortcut to transmit the extracted path rule to theelectronic device.

In operation 4105, an electronic device 4101 (e.g., the processor 2550of FIG. 26 ) may receive the voice input from a user.

According to an embodiment, the electronic device may receive the voiceinput from the user via a microphone.

In operation 4107, the electronic device 4101 (e.g., the processor 2550of FIG. 26 ) may transmit the received voice input to a server 4103(e.g., the intelligent server 2600).

According to an embodiment, the electronic device 4101 may transmit thevoice input to the server 4103 via a wireless communication circuit.

In operation 4109, the server 4103 (e.g., the intelligent server 2600)may confirm text data corresponding to the voice input.

According to an embodiment, the server 4103 may use an ASR technique toconvert the voice input into text data. For example, the server 4103 mayinclude an ASR module (e.g., the ASR module 2610) to convert the voiceinput to the text data.

In operation 4111, the server 4103 (e.g., the intelligent server 2600)may determine whether the voice input corresponds to a pre-storedcommand shortcut based on the text data.

According to an embodiment, the server 4103 may determine whether thetext data is a command shortcut set as a macro. The command shortcut maybe stored in the database of the server in advance. For example, thecommand shortcut may be mapped with the representative dialog requestingto perform any one function (task) of the electronic device, and may bestored in advance in the database together with the representativedialog. For example, the representative dialog may be mapped to the pathrule including the information of the sequence of the states of theelectronic device for performing a function.

According to an embodiment, when the voice input corresponds to thepre-stored command shortcut, the server 4103 (e.g., the intelligentserver 2600) may confirm the representative dialog corresponding to thecommand shortcut in operation 4113.

In operation 4115, the server 4103 (e.g., the intelligent server 2600)may extract the path rule using the representative dialog.

According to an embodiment, the server 4103 may confirm a preset pathrule corresponding to the representative dialog.

According to another embodiment, the server 4103 may include an NLUmodule (e.g., the NLU module 2620) and a path planner module (e.g., thepath planner module 2630) to generate the path rule corresponding to therepresentative dialog. Since the detailed operation for generating(extracting) the path rule has been described in the above-describedembodiments, a detailed description thereof will be omitted.

In operation 4117, the server 4103 (e.g., the intelligent server 2600)may transmit the path rule to the electronic device 4101 (e.g., theprocessor 2550 of FIG. 26 ).

According to an embodiment, the server 4103 may transmit the path ruleto the electronic device 4101 via a wireless communication circuit.

In operation 4119, the electronic device 4101 (e.g., the processor 2550of FIG. 26 ) may perform an operation corresponding to the path rule.

According to an embodiment, the electronic device 4101 may include anexecution manager module (e.g., the execution manager module 2553) toexecute the operation corresponding to the path rule.

FIG. 42 is a flowchart illustrating an operation of executing thefunction of an electronic device (e.g., the user terminal 2500)according to various embodiments.

Referring to FIG. 42 , the electronic device may store a representativedialog set to perform any one function of the electronic device and acommand shortcut mapped to the representative dialog, and therepresentative dialog may correspond to a path rule includinginformation of a sequence of states of the electronic device. Whenreceiving a command shortcut as a text input corresponding to thecommand shortcut from a user, the electronic device may execute afunction corresponding to the command shortcut via communication withthe server.

In operation 4205, an electronic device 4201 (e.g., the processor 2550of FIG. 26 ) may receive the text input from the user.

According to an embodiment, the electronic device may receive the textinput from the user via a touch screen display.

In operation 4207, the electronic device 4201 (e.g., processor 2550 ofFIG. 26 ) may determine whether the text input corresponds to apre-stored command shortcut.

According to an embodiment, the electronic device 4201 may determinewhether the text input is a command shortcut set as a macro. The commandshortcut may be stored in advance in the database of the electronicdevice. For example, the command shortcut may be mapped with therepresentative dialog requesting to perform a function (task) of theelectronic device, and may be pre-stored in the database together withthe representative dialog. For example, the representative dialog may bemapped with a path rule including information of a sequence of states ofthe electronic device for performing a function.

According to an embodiment, when the text input corresponds to thepre-stored command shortcut, the electronic device 4201 may confirm therepresentative dialog corresponding to the short-instruction inoperation 4209.

In operation 4211, the electronic device 4201 (e.g., the processor 2550of FIG. 26 ) may transmit the confirmed representative dialog to theserver 4203 (e.g., the intelligent server 2600).

According to an embodiment, the electronic device 4201 may transmit therepresentative dialog to the server 4203 via a wireless communicationcircuit. In operation 4213, the server 4203 (e.g., the intelligentserver 2600) may extract the path rule using the representative dialog.

According to an embodiment, the server 4203 may confirm a preset pathrule corresponding to the representative dialog.

According to another embodiment, the server 4203 may include an NLUmodule (e.g., the NLU module 2620) and a path planner module (e.g., thepath planner module 2630) to generate the path rule corresponding to therepresentative dialog. Since the detailed operation for generating(extracting) the path rule has been described in the above-describedembodiments, a detailed description thereof will be omitted.

In operation 4215, the server 4203 (e.g., the intelligent server 2600)may transmit the path rule to an electronic device 4201 (e.g., theprocessor 2550 of FIG. 26 ).

According to an embodiment, the server 4203 may transmit the path ruleto the electronic device 4201 via a wireless communication circuit.

In operation 4217, the electronic device 4201 (e.g., the processor 2550of FIG. 26 ) may perform an operation corresponding to the path rule.

According to an embodiment, the electronic device 4001 may include anexecution manager module (e.g., the execution manager module 2553) toperform the operation corresponding to the path rule.

FIG. 43 is a flowchart illustrating an operation of executing thefunction of an electronic device (e.g., the user terminal 2500)according to various embodiments.

Referring to FIG. 43 , a server may store a representative dialog set toperform any one function of the electronic device and a command shortcutmapped to the representative dialog, and the representative dialog maycorrespond to a path rule including information of a sequence of statesof the electronic device. When receiving a text input corresponding tothe command shortcut from the electronic device, the server may extracta path rule corresponding to the command shortcut to transmit theextracted path rule to the electronic device.

In operation 4305, an electronic device 4301 (e.g., the processor 2550of FIG. 26 ) may receive the text input from a user.

According to an embodiment, the electronic device may receive the textinput from the user via a touch screen display.

In operation 4307, the electronic device 4301 (e.g., the processor 2550of FIG. 26 ) may transmit the received text input to a server 4303(e.g., the intelligent server 2600).

According to an embodiment, the electronic device 4301 may transmit thetext input to the server 4303 via a wireless communication circuit.

In operation 4309, the server 4303 (e.g., the intelligent server 2600)may determine whether the text input corresponds to a pre-stored commandshortcut.

According to an embodiment, the server 4303 may determine whether thetext input is a command shortcut set as a macro. The command shortcutmay be stored in advance in the database of the server. For example, thecommand shortcut may be mapped with the representative dialog requestingto perform a function (task) of the electronic device, and may bepre-stored in the database together with the representative dialog. Forexample, the representative dialog may be mapped with a path ruleincluding information of a sequence of states of the electronic devicefor performing a function.

According to an embodiment, when the text input corresponds to thepre-stored command shortcut, the server 4303 may confirm arepresentative dialog corresponding to the command shortcut in operation4311.

In operation 4313, the server 4303 (e.g., intelligent server 2600) mayextract the path rule using the representative dialog.

According to an embodiment, the server 4303 may confirm a preset pathrule corresponding to the representative dialog.

According to another embodiment, the server 4303 may include an NLUmodule (e.g., the NLU module 2620) and a path planner module (e.g., thepath planner module 2630) to generate the path rule corresponding to therepresentative dialog. Since the detailed operation for generating(extracting) the path rule has been described in the above-describedembodiments, a detailed description thereof will be omitted.

In operation 4315, the server 4303 (e.g., the intelligent server 2600)may transmit the path rule to the electronic device 4301 (e.g., theprocessor 2550 of FIG. 26 ).

According to an embodiment, the server 4303 may transmit the path ruleto the electronic device 4301 via a wireless communication circuit.

In operation 4317, the electronic device 4301 (e.g., the processor 2550in FIG. 26 ) may perform an operation corresponding to the path rule.

According to an embodiment, electronic device 4301 may include anexecution manager module (e.g., execution manager module 2553) toperform the operation corresponding to the path rule.

FIG. 44 is a diagram illustrating the registration of a command shortcutfor executing the function of an electronic device (e.g., the userterminal 2500) according to various embodiments.

Referring to FIG. 44 , at the time of a user input for executing any onefunction of the electronic device, the electronic device may display ascreen for registration of a command shortcut having a macro function soas to simply execute the function. The electronic device may registerthe command shortcut of the function as a macro based on the user input.For example, the electronic device may receive a voice input of a usercorresponding to a representative dialog for executing a specificfunction, and may execute a function corresponding to the voice input.When the execution of the function is completed, the electronic devicemay cause a user to input a simple command shortcut (also referred to asa macro) in place of the above-described representative dialog so thatthe above-described function can be performed through the correspondingcommand shortcut later.

According to an embodiment, as shown in FIG. 44A, the electronic devicemay receive a voice input of the user. For example, the electronicdevice may receive the voice input of the user such as “Show me phototaken yesterday in gallery” that is the representative dialog forexecuting a function of displaying a photo taken and stored yesterday onthe screen from the gallery application of the electronic device.

According to an embodiment, depending on the execution of the functioncorresponding to the voice input, the electronic device may display auser interface (UI) 4403 for guiding the registration of the commandshortcut (e.g., a macro) for the function on the screen, as shown inFIG. 44B. For example, the UI 4403 may include the execution result ofthe function. Referring to FIG. 44B, the UI 4403 may include a commandshortcut registration menu 4401 (e.g., a button) for displaying acommand shortcut registration screen that allows the received commandshortcut of the function to be registered as a macro, and a naturallanguage phrase indicating a result of execution of the function, suchas “Okay, I found 2 pictures,” as depicted.

According to an embodiment, at the time of a touch input of the user forthe command shortcut registration menu 4401 or a voice input of the usersuch as “Add command shortcut” that is a command corresponding to thecommand shortcut registration menu 4401, the electronic device mayexecute the command shortcut registration screen as shown in FIG. 44C.Referring to FIG. 44C, the command shortcut registration screen maydisplay a button 4405 for allowing the registration of the commandshortcut to be executed, together with a notice such as “Tell me commandshortcut to be used instead of long command” that informs the user of amethod of setting the command shortcut to be designated as the macro ofthe function. For example, the user may touch the button 4405 to utterthe command shortcut desired to be designated so that the electronicdevice may designate the command shortcut as the macro of the function.In addition, the command shortcut registration screen may include “Showme photo taken yesterday in gallery” that is a representative dialog ofa function to designating the command shortcut as shown in FIG. 44C.

In the above-described embodiment of FIG. 44 , it has been describedthat the user utters the command shortcut desired to be designated sothat the electronic device may designate the command shortcut as themacro of the function. However, according to another embodiment, byperforming a text input of the command shortcut desired to be designatedby the user, the electronic device may designate the command shortcut asthe macro of the function.

FIG. 45 is a diagram illustrating the registration of a command shortcutfor executing the function of an electronic device (e.g., the userterminal 2500) according to various embodiments.

Referring to FIG. 45 , the electronic device may display a command listreceived from a user on a screen, and the command list may include amenu for allowing a command shortcut to be added.

According to an embodiment, the electronic device may display a command4505 received from the user and a list of an item 4501 including anexecution result 4503 of a function corresponding to the command on ahistory screen as shown in FIG. 45A. A corresponding setting menu 4507may be displayed in each of the commands 4505 included in the historyscreen. When the user selects the setting menu 4507, a user UI may begenerated, and the generated UI may include a command shortcut additionmenu 4509 for allowing the command shortcut (macro) to be added and ahistory deletion menu 4511 for allowing a user history item to bedeleted. When the command shortcut addition menu 4509 is selectedaccording to a user input, the electronic device may execute the commandshortcut registration screen as shown FIG. 45B. Referring to FIG. 45B,the command shortcut registration screen may display a button 4513 forallowing the command shortcut registration to be executed, together witha notice such as “Tell me command shortcut to be used instead of longcommand” that informs the user of a method of setting the commandshortcut to be designated as the macro of the function. For example, theuser may touch the button 4513 to utter the command shortcut desired tobe designated so that the electronic device may designate the commandshortcut as the macro of the function. In addition, the command shortcutregistration screen may include “Send photo just taken to Kyung-soo viaKakao Talk” that is the representative dialog of the function ofdesignating the command shortcut as shown in FIG. 45B.

In the above-described embodiment of FIG. 45 , it has been describedthat the user utters the command shortcut desired to be designated sothat the electronic device may designate the command shortcut as themacro of the corresponding function. However, according to anotherembodiment, by performing a text input of the command shortcut desiredto be designated by the user, the electronic device may designate thecommand shortcut as the macro of the function.

FIG. 46 is a diagram illustrating the registration of a command shortcutfor executing the function of an electronic device (e.g., the userterminal 2500) according to various embodiments.

Referring to FIG. 46 , the electronic device may collect and displaycommand shortcuts which have been set as macros to be stored. Forexample, the electronic device may display a list of the commandshortcuts set as the macros, and may perform operations related toadditional registration, modification, and deletion of the commandshortcuts.

Referring to FIG. 46A, the electronic device may predict functions to beperformed by a user in the electronic device, and may display hints 4603corresponding to utterance commands capable of performing the functions.In addition, when a touch input of the user is performed and then anutterance input of the user is performed, the electronic device maydisplay a button 4605 for allowing a function corresponding to theutterance to be executed. In addition, the electronic device may displaya menu (MACRO) 4601 for allowing the command shortcut corresponding toeach of the hints 4603 to be set.

When the menu 4601 is selected according to the user input, theelectronic device may display a list screen including a list 4607 of thecommand shortcuts which have been set as macros to be stored as shown inFIG. 46B. In addition, the list screen may include an additionalregistration button (ADD) 4609 for allowing the command shortcuts to beadditionally registered.

When receiving a user input for the additional registration button 4609,the electronic device may display an additional registration screen forallowing the command shortcut corresponding to each of the hintsprovided by the electronic device to be registered as shown in FIG. 46C.Referring to FIG. 46C, the additional registration screen may include afirst item 4611 for allowing the command shortcut to be input, a seconditem 4612 for allowing the hint to be retrieved, and a third item 4613for displaying a hint list. The electronic device may combine thecommand shortcut input to the first item 4611 and the hint selected bythe user from the second item 4612 or the third item 4613 to generate amacro. For example, the electronic device may generate the commandshortcut input to the first item 4611 as a macro of a function ofexecuting the hint selected by the user from the second item 4612 or thethird item 4613

FIG. 47 is a diagram illustrating the registration of a command shortcutfor executing the function of an electronic device (e.g., the userterminal 2500) according to various embodiments.

According to an embodiment, when a hint is used to set a commandshortcut as a macro, the command shortcut may be generated by modifyinga parameter as shown in FIG. 47 . For example, a hint such as “Call toMom” may be provided in consideration of history, TOP information, orthe like, but a function for which a macro is desired to be generated bya user may be call to John, and thereby “Call to John” may be generatedas the hint corresponding to the function. Although a user can directlywrite the entire dialog that is mapped with the desired commandshortcut, when the hint is used, a precise representative dialog mappedwith the corresponding function may be mapped, so that the operation ofthe server (e.g., the NLU module, the PP module, etc.) can be moreprecisely performed. For example, when the parameter is modified, thisdoes not affect the structure of the representative dialog, so that itis possible to modify the parameter, thereby increasing the accuracy ofthe operations corresponding to the path rules and increasing thevariety of operations that the user can perform.

Referring to FIG. 47A, the electronic device may predict functions to beperformed in the electronic device by the user, and may display hints4703 corresponding to utterance commands capable of executing thefunctions. In addition, the electronic device may display a menu (MACRO)4701 that allows the user to set a command shortcut corresponding toeach of the hints 4703.

When the menu 4701 is selected according to the user input, theelectronic device may display a list screen including a list 4707 ofcommand shortcuts which have been set and stored as macros as shown inFIG. 47B, including “OMG” 4709.

When any one command shortcuts 4707 included in the list of commandshortcuts 4707 is selected by the user, the electronic device maydisplay a parameter screen for allowing the parameter of the selectedcommand shortcut 4707 to be modified as shown in FIG. 47C. Referring toFIG. 47C, when “OMG” macro 4709 is selected, a parameter modificationscreen for modifying the “OMG” macro may include a first item 4711 inwhich the command shortcut 4707 is displayed, and a second item 4713 inwhich a representative dialog is displayed in order to modify theparameter of the representative dialog corresponding to the commandshortcut 4707. A parameter part of the representative dialog shown inthe second item 4713 may indicate that the parameter part can bemodified, such as changing the color of the text, underlining, or thelike. In addition, the parameter modification screen may include a thirditem 4715 for displaying the hint list.

FIG. 48 is a diagram illustrating a command shortcut execution screen ofan electronic device (e.g., the user terminal 2500) according to variousembodiments.

According to an embodiment, in accordance with the above-describedembodiments, a command shortcut may be mapped with a representativedialog to be registered as a macro. When a user inputs a commandshortcut, the electronic device may display a function execution resultaccording to the command shortcut.

For example, according to the above-described embodiments, the user maycause a command shortcut such as “Today” to be mapped with arepresentative dialog such as “Today's weather” and registered as amacro. When the user utters “Today”, the electronic device may displayan execution result screen of the command shortcut as shown in FIG. 48 .Referring to FIG. 48 , the execution result screen may include “Today'sweather” 4805 that is an execution result of a function corresponding to“Today's weather” which is a representative dialog 4801. In addition,the execution result screen may further include “Today” which is acommand shortcut 4801 and “Today's weather” which is a representativedialog 4803 corresponding to the “Today” command shortcut 4801.

According to various embodiments, although not shown, in order toactivate the use of the macro function described above, the server mayprovide reward services such as providing points that can be used whenusing manufacturer's service, depending on the macros which have beenregistered and stored through the electronic device. For example, it ispossible to improve the frequency of use of the macro function and toprovide various services to a user (electronic device) having a highfrequency of use.

According to various embodiments, an electronic device (e.g., the userterminal 2500) may include a housing; a touch screen display (e.g., thedisplay 2520) that is located inside the housing and is exposed througha first portion of the housing; a microphone (e.g., the microphone 288)that is located inside the housing and is exposed through a secondportion of the housing; at least one speaker (e.g., the speaker 2530)that is located inside the housing and is exposed through a thirdportion of the housing; a wireless communication circuit (e.g., thecommunication module 220) that is located inside the housing; aprocessor (e.g., the processor 2550) that is located inside the housingand is electrically connected to the touch screen display, themicrophone, the speaker, and the wireless communication circuit; and amemory (e.g., the memory 230) that is located inside the housing and iselectrically connected to the processor, such that when the memory isperformed, the processor receives a first user input via at least one ofthe touch screen display and the microphone, the first user inputincluding a request to perform a task using at least one of theelectronic device and an external device, transmits first dataassociated with the first user input to an external server (e.g., theintelligent server 2600) via the wireless communication circuit,receives a first response from the external server via the wirelesscommunication circuit, the first response including informationassociated with the first user input and/or sequence information ofstates of the electronic device for performing at least a portion of thetask, receives a second user input for assigning an utterance and/or auser operation via the touch screen display as a request to perform thetask, and stores instructions causing to transmit second data associatedwith the second user input to the external server via the communicationcircuit.

According to various embodiments, the instructions may cause theprocessor to receive a third user input via the at least one of thetouch screen display and the microphone after the second data istransmitted, the third user input including the utterance and/or theuser operation, to transmit third data associated with the third userinput to the external server via the wireless communication circuit, toreceive a second response from the external server via the wirelesscommunication circuit, the second response including the sequenceinformation of the states of the electronic device for performing atleast a portion of the task, and to allow the electronic device to havethe sequence of the states to perform the at least a portion of thetask.

According to various embodiments, the instructions may cause theprocessor to store a log associated with the first user input receivedvia the touch screen display, and to transmit the log as a part of thefirst data.

According to various embodiments, the second user input may include anutterance personalized by a user and/or a graphic user interfaceselected by the user.

According to various embodiments, the memory may store at least onerepresentative dialog that corresponds to the sequence of the states ofthe electronic device and is a request to perform the task, and acommand shortcut that is mapped with the representative dialog, and theinstructions may cause the processor to transmit the second data that isthe representative dialog corresponding to a voice to the externalserver, id the second user input is the voice received via themicrophone.

According to various embodiments, the instructions may cause theprocessor to transmit the voice to the external server via the wirelesscommunication circuit, to receive text data corresponding to the voicefrom the external server via the wireless communication circuit, and toidentify the representative dialog matched with the command shortcut ifthe text data corresponds to the command shortcut.

According to various embodiments, the instructions may cause theprocessor to receive the sequence information of the states of theelectronic device that corresponds to the second data and is used toperform the at least a portion of the task via the wirelesscommunication circuit after the second data is transmitted, and toperform the at least a portion of the task.

According to various embodiments, the memory may store at least onerepresentative dialog that corresponds to the sequence of the states ofthe electronic device and is a request to perform the task and a commandshortcut that is mapped with the representative dialog, and theinstructions may cause the processor to transmit the second data that isthe representative dialog corresponding to a text input to the externalserver, when the second user input is the text input received via thetouch screen display.

According to various embodiments, the instructions may cause theprocessor to identify the representative dialog matched with the commandshortcut if the text input corresponds to the command shortcut.

According to various embodiments, a method of executing an operation ofan electronic device (e.g., the user terminal 2500) may include:receiving a first user input, such that the first user input includes arequest to perform a task using at least one of the electronic device oran external device; transmitting first data associated with the firstuser input to an external server (e.g., the intelligent server 2600);receiving a first response from the external server, such that the firstresponse includes information associated with the first user inputand/or sequence information of states of the electronic device forperforming at least a portion of the task; receiving a second user inputfor assigning an utterance and/or a user operation via a touch screendisplay of the electronic device as a request to perform the task; andtransmitting second data associated with the second user input to theexternal server.

According to various embodiments, the method may further includereceiving a third user input after the second data is transmitted, suchthat the third user input includes the utterance and/or the useroperation; transmitting third data associated with the third user inputto the external server; receiving a second response from the externalserver, such that the second response includes the sequence informationof the states of the electronic device for performing the at least aportion of the task; and allowing the electronic device to have thesequence of the states to perform the at least a portion of the task.

According to various embodiments, the method may further include storinga log associated with the first user input received via the touch screendisplay of the electronic device, and transmitting the log as a part ofthe first data.

According to various embodiments, the second user input may include anutterance personalized by a user and/or a graphic user interfaceselected by the user.

According to various embodiments, the method may further include storingat least one representative dialog that corresponds to the sequence ofthe states of the electronic device and is a request to perform thetask, and a command shortcut that is mapped with the representativedialog, and transmitting the second data that is the representativedialog corresponding to a voice to the external server, if the seconduser input is the voice received via the microphone.

According to various embodiments, the method may further includetransmitting the voice to the external server via the wirelesscommunication circuit, receiving text data corresponding to the voicefrom the external server via the wireless communication circuit, andidentifying the representative dialog matched with the command shortcutif the text data corresponds to the command shortcut.

According to various embodiments, a machine-readable storage mediumrecording a program to perform a method of executing an operation of anelectronic device, the method comprising: receiving a first user input,such that the first user input includes a request to perform a taskusing at least one of the electronic device or an external device;transmitting first data associated with the first user input to anexternal server; receiving a first response from the external server,such that the first response includes information associated with thefirst user input and/or sequence information of states of the electronicdevice for performing at least a portion of the task; receiving a seconduser input for assigning an utterance and/or a user operation via atouch screen display of the electronic device as the request to performthe task; and transmitting second data associated with the second userinput to the external server.

According to various embodiments, an electronic device (e.g., the userterminal 2500) may include: a housing; a touch screen display (e.g., thedisplay 2520) that is located inside the housing and is exposed througha first portion of the housing; a microphone (e.g., the microphone 288)that is located inside the housing and is exposed through a secondportion of the housing; at least one speaker (e.g., the speaker 2530)that is located inside the housing and is exposed through a thirdportion of the housing; a wireless communication circuit (e.g., thecommunication module 220) that is located inside the housing; aprocessor (e.g., the processor 2550) that is located inside the housingand is electrically connected to the touch screen display, themicrophone, the speaker, and the wireless communication circuit; and amemory (e.g., the memory 230) that is located inside the housing and iselectrically connected to the processor, such that when the memory isperformed, the processor receives a first user input via at least one ofthe touch screen display and the microphone, the first user inputincluding a request to perform a task using at least one of theelectronic device and an external device, transmits first dataassociated with the first user input to an external server via thewireless communication circuit, acquires a first response based on thefirst data associated with the first user input, the first responseincluding information associated with the first user input and/orsequence information of states of the electronic device for performingat least a portion of the task, receives a second user input forassigning an utterance and/or a user operation via the touch screendisplay as a request to perform the task, and acquires a second responsebased on second data associated with the second user input via thecommunication circuit, the second response storing instructionsincluding the sequence information of the states of the electronicdevice for performing the at least a portion of the task.

According to various embodiments, the instructions may cause theprocessor to receive a third user input via the at least one of thetouch screen display and the microphone after the second data isacquired, the third user input including the utterance and/or the useroperation, to receive a third response based on third data associatedwith the third user input via the wireless communication circuit, thethird response including the sequence information of the states of theelectronic device for performing the at least a portion of the task, andto allow the electronic device to have the sequence of the states toperform the at least a portion of the task.

According to various embodiments, the instructions may cause theprocessor to store a log associated with the first user input receivedvia the touch screen display, and to transmit the log as a part of thefirst data.

According to various embodiments, the second user input may include anutterance personalized by a user and/or a graphic user interfaceselected by the user.

The term “module” as used herein may include a unit including hardware,software, or firmware, and may, for example, be used interchangeablywith the term “logic”, “logical block”, “component”, “circuit”, or thelike. The “module” may be an integrated component, or a minimum unit forperforming one or more functions or a part thereof. The “module” may bemechanically or electronically implemented and may include, for example,an Application-Specific Integrated Circuit (ASIC) chip, aField-Programmable Gate Arrays (FPGA), or a programmable-logic device,which has been known or are to be developed in the future, forperforming certain operations. At least some of devices (e.g., modulesor functions thereof) or methods (e.g., operations) according to variousembodiments may be implemented by an instruction which is stored acomputer-readable storage medium (e.g., the memory 130) in the form of aprogram module. The instruction, when executed by a processor (e.g., theprocessor 120), may cause the one or more processors to execute thefunction corresponding to the instruction. The computer-readable storagemedium may include a hard disk, a floppy disk, a magnetic medium (e.g.,a magnetic tape), an Optical Media (e.g., CD-ROM, DVD), aMagneto-Optical Media (e.g., a floptical disk), an inner memory, etc.The instruction may include a code made by a complier or a code that canbe executed by an interpreter. The programming module according to thepresent disclosure may include one or more of the aforementionedelements or may further include other additional elements, or some ofthe aforementioned elements may be omitted. Operations performed by amodule, a programming module, or other elements according to variousembodiments may be executed sequentially, in parallel, repeatedly, or ina heuristic manner. At least some operations may be executed accordingto another sequence, may be omitted, or may further include otheroperations.

What is claimed is:
 1. A portable communication device comprising: amemory; a microphone; a touchscreen display; and at least one processorconfigured to: after a function of an application is executed, display,via the touchscreen display, an indication of the function of theapplication for registering a shortcut for executing the function of theapplication using a voice command, wherein the indication is displayedfor the shortcut registration based at least in part on a determinationthat the function is executable based at least in part on one or morevoice command; receive, via the touchscreen display, a first input withrespect to the indication; based at least in part on the first input,display, via the touchscreen display, a text associated with thefunction; and receive a user text input while the text is displayed,wherein at least part of the user text input is registered as a voicecommand to be used as a shortcut to execute the function of theapplication.
 2. The portable communication device of claim 1, whereinthe indication includes a text of a graphical indication.
 3. Theportable communication device of claim 1, wherein the at least oneprocessor is configured to generate the text such that a name of theapplication is included as at least part of the text.
 4. The portablecommunication device of claim 1, wherein the at least one processor isconfigured to transmit at least part of the user input to a serverexternal to the portable communication device, such that the serverregisters the at least part of the user input as the voice command toexecute the function.
 5. The portable communication device of claim 1,wherein the portable communication device further includes acommunication circuit, and wherein the at least one processor furtherconfigured to transmit the voice command received via the microphone toa server so that the server determines whether a first textcorresponding to the user voice command substantially identical with asecond text, stored in the server, corresponding to the user text input.6. The portable communication device of claim 1, wherein the at leastone processor is configured to execute a plurality of functionsincluding the function and an another function based at least in part onreception of the registered voice command.
 7. The portable communicationdevice of claim 1, wherein the at least one processor is configured totransmit a voice input to a server external to the portablecommunication device, such that the server determines that the voiceinput corresponds to the registered voice command.
 8. A portablecommunication device comprising: a microphone; a touchscreen display;and at least one processor configured to: after executing of a firstfunction is terminated, display, via the touchscreen display, a text ora graphical indication based at least in part on a determination that afirst function is executable based at least in part on one or more voicecommand being received via the microphone, the text including one ormore words related to the first function; receive a first touch inputwith respect to the text or the graphical indication; and afterreceiving the first touch input, receive a user text input forregistering a shortcut, wherein the user text input is registered as avoice command to be used to execute the first function.
 9. The portablecommunication device of claim 8, wherein the at least one processor isconfigured to: receive a voice input after at least part of the usertext input is registered as the voice command; and based at least inpart on a determination that the voice input corresponds to the voicecommand, execute the first function at the portable communicationdevice.
 10. The portable communication device of claim 9, wherein thevoice input includes a different word from the one or more wordsincluded in the text.
 11. The portable communication device of claim 8,wherein the at least one processor is configured to generate the textsuch that at least part of the text includes a name of a specifiedapplication corresponding to the first function.
 12. The portablecommunication device of claim 8, wherein the at least one processor isconfigured to transmit the at least part of the user text input to aserver external to the portable communication device, such that theserver registers the at least part of the user text input as the voicecommand to be used to execute the first function.
 13. The portablecommunication device of claim 8, further comprising a memory to store afirst application, wherein the at least one processor is configured tostore, in the memory, the at least part of the user text input as thevoice command to be used to execute the first function.
 14. The portablecommunication device of claim 9, wherein the first function isexecutable using a plurality of applications including a firstapplication and a second application, and wherein the at least oneprocessor is configured to: as at least part of the executing of thefirst function, execute the first application based at least in part onreception of the registered voice command, and after the firstapplication is executed, execute the second application based at leastin part on at least one of first data generated according to an anotheruser text input or second data included in the voice input.
 15. Aportable communication device comprising: a memory; a microphone; atouchscreen display; and at least one processor configured to: after afunction of an application is executed, display, via the touchscreendisplay, an indication of the function of the application forregistering a shortcut for executing the function of the applicationusing a voice command, wherein the indication is displayed for theshortcut registration based at least in part on a determination that thefunction is executable based at least in part on one or more voicecommand; receive, via the touchscreen display, a first input withrespect to the indication; based at least in part on the first input,display, via the touchscreen display, a text associated with thefunction; and receive a user voice input while the text is displayed,wherein at least part of the user voice input is registered as a voicecommand to be used as a shortcut to execute the function of theapplication.
 16. The portable communication device of claim 15, whereinthe indication includes a text of a graphical indication.
 17. Theportable communication device of claim 15, wherein the at least oneprocessor is configured to generate the text such that a name of theapplication is included as at least part of the text.
 18. The portablecommunication device of claim 15, wherein the at least one processor isconfigured to transmit at least part of the user input to a serverexternal to the portable communication device, such that the serverregisters the at least part of the user input as the voice command toexecute the function.
 19. The portable communication device of claim 15,wherein the portable communication device further includes acommunication circuit, and wherein the at least one processor furtherconfigured to transmit the voice command received via the microphone toa server so that the server determines whether the a first textcorresponding to the voice command substantially identical with a secondtext, stored in the server, corresponding to the user voice input. 20.The portable communication device of claim 15, wherein the at least oneprocessor is configured to execute a plurality of functions includingthe function and an another function based at least in part on receptionof the registered voice command.